[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:07:17 dist-test-slave-s3n3 krb5kdc[6162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:07:17 dist-test-slave-s3n3 krb5kdc[6162](info): set up 2 sockets
Sep 24 10:07:17 dist-test-slave-s3n3 krb5kdc[6162](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 10:07:18.223568 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:07:18 dist-test-slave-s3n3 krb5kdc[6162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:07:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:07:18.413269 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:44405
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:44405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:18.753760 6175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:18.754072 6175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:18.754272 6175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:18.761679 6175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:18.761801 6175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:18.761898 6175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:18.761988 6175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:18.762028 6175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:18.772284 6175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:44405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:44405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:18.773916 6175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:18.775398 6175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:18.789098 6180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:18.789178 6181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:18.789132 6183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:18.789817 6175 server_base.cc:1034] running on GCE node
I20260924 10:07:18.790639 6175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:18.792305 6175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:18.793589 6175 hybrid_clock.cc:648] HybridClock initialized: now 1790244438793539 us; error 67 us; skew 500 ppm
I20260924 10:07:18.794343 6175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:18 dist-test-slave-s3n3 krb5kdc[6162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244438, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:18.798410 6175 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:07:18.800679 6175 webserver.cc:460] Webserver started at http://127.6.2.254:34133/ using document root <none> and password file <none>
I20260924 10:07:18.801554 6175 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:18.801719 6175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:18.802083 6175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:18.805534 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "9daa4e96ad9b4b8f84f98d0e14e5dfe9"
format_stamp: "Formatted at 2026-09-24 10:07:18 on dist-test-slave-s3n3"
server_key: "e3c0191bdd9606cd3414377d78941b97"
server_key_iv: "f6d410f37e1ac266384916a7d8eb7fb5"
server_key_version: "encryptionkey@0"
I20260924 10:07:18.806371 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "9daa4e96ad9b4b8f84f98d0e14e5dfe9"
format_stamp: "Formatted at 2026-09-24 10:07:18 on dist-test-slave-s3n3"
server_key: "e3c0191bdd9606cd3414377d78941b97"
server_key_iv: "f6d410f37e1ac266384916a7d8eb7fb5"
server_key_version: "encryptionkey@0"
I20260924 10:07:18.813475 6175 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260924 10:07:18.818454 6190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:18.821038 6175 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 10:07:18.821552 6175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "9daa4e96ad9b4b8f84f98d0e14e5dfe9"
format_stamp: "Formatted at 2026-09-24 10:07:18 on dist-test-slave-s3n3"
server_key: "e3c0191bdd9606cd3414377d78941b97"
server_key_iv: "f6d410f37e1ac266384916a7d8eb7fb5"
server_key_version: "encryptionkey@0"
I20260924 10:07:18.821923 6175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:18.854954 6175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:18.860221 6175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:18.860528 6175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:18.880434 6175 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:44405
I20260924 10:07:18.880478 6241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:44405 every 8 connection(s)
I20260924 10:07:18.881903 6175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:18.884392 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 6175
I20260924 10:07:18.884749 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:07:18.886790 6242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:18.887377 6155 external_mini_cluster.cc:1428] Setting key c9ea3331f7bc2ce71e3e1d5752be31bd
I20260924 10:07:18.900596 6242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9: Bootstrap starting.
I20260924 10:07:18.904052 6242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:18.905233 6242 log.cc:824] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:18.908335 6242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9: No bootstrap required, opened a new log
I20260924 10:07:18.917451 6242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9daa4e96ad9b4b8f84f98d0e14e5dfe9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44405 } }
I20260924 10:07:18.918030 6242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:18.918169 6242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9daa4e96ad9b4b8f84f98d0e14e5dfe9, State: Initialized, Role: FOLLOWER
Sep 24 10:07:18 dist-test-slave-s3n3 krb5kdc[6162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:18.918999 6242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9daa4e96ad9b4b8f84f98d0e14e5dfe9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44405 } }
I20260924 10:07:18.919415 6242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:18.919607 6242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:18.919816 6242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:18.923504 6242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9daa4e96ad9b4b8f84f98d0e14e5dfe9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44405 } }
I20260924 10:07:18.924382 6242 leader_election.cc:302] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9daa4e96ad9b4b8f84f98d0e14e5dfe9; no voters:
I20260924 10:07:18.925109 6242 leader_election.cc:288] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:18.925465 6247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:18.926748 6247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [term 1 LEADER]: Becoming Leader. State: Replica: 9daa4e96ad9b4b8f84f98d0e14e5dfe9, State: Running, Role: LEADER
I20260924 10:07:18.927578 6247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9daa4e96ad9b4b8f84f98d0e14e5dfe9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44405 } }
I20260924 10:07:18.927975 6242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:18.930625 6248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9daa4e96ad9b4b8f84f98d0e14e5dfe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9daa4e96ad9b4b8f84f98d0e14e5dfe9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44405 } } }
I20260924 10:07:18.931213 6248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:18.931731 6249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9daa4e96ad9b4b8f84f98d0e14e5dfe9. Latest consensus state: current_term: 1 leader_uuid: "9daa4e96ad9b4b8f84f98d0e14e5dfe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9daa4e96ad9b4b8f84f98d0e14e5dfe9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44405 } } }
I20260924 10:07:18.932045 6249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:18.932283 6252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:18.933531 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:18.892129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36960 (local address 127.6.2.254:44405)
0924 10:07:18.893164 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:18.893182 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:18.894339 (+ 1157us) server_negotiation.cc:407] Connection header received
0924 10:07:18.903915 (+ 9576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:18.903963 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:18.904655 (+ 692us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:18.905242 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:18.909299 (+ 4057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:18.910698 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:18.912527 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:18.913874 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:18.914149 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:18.921137 (+ 6988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:18.921149 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:18.921173 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:18.921229 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:18.926198 (+ 4969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:18.927709 (+ 1511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:18.927721 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:18.927728 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:18.927972 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:18.928589 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:18.928594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:18.928597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:18.929490 (+ 893us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:18.931202 (+ 1712us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:18.931550 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 10:07:18.932044 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":427,"threads_started":1}
I20260924 10:07:18.939342 6252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:18.955705 6252 catalog_manager.cc:1347] Generated new cluster ID: 0b6dda253c9942b4aec6f827c55e8b2d
I20260924 10:07:18.955845 6252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:18.975893 6252 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:18.977337 6252 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:18.992354 6252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9daa4e96ad9b4b8f84f98d0e14e5dfe9: Generated new TSK 0
I20260924 10:07:18.993265 6252 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:07:19.039903 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:44405
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:19.380735 6268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:19.381044 6268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:19.381203 6268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:19.388816 6268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:19.388944 6268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:19.389142 6268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:07:19.401430 6268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:44405
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:19.402789 6268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:19.404603 6268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:19.423542 6273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:19.423580 6274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:19.423602 6276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:19.424530 6268 server_base.cc:1034] running on GCE node
I20260924 10:07:19.425217 6268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:19.426309 6268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:19.427561 6268 hybrid_clock.cc:648] HybridClock initialized: now 1790244439427465 us; error 75 us; skew 500 ppm
I20260924 10:07:19.428069 6268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:19 dist-test-slave-s3n3 krb5kdc[6162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:19.431531 6268 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:07:19.433180 6268 webserver.cc:460] Webserver started at http://127.6.2.193:32883/ using document root <none> and password file <none>
I20260924 10:07:19.433892 6268 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:19.434053 6268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:19.434367 6268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:19.437438 6268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "e7fc2d2cadc840d9b4a877f2ad84c6a9"
format_stamp: "Formatted at 2026-09-24 10:07:19 on dist-test-slave-s3n3"
server_key: "3ebd2356eeb3b4d89e487dc3422f6057"
server_key_iv: "f76ce2ba0630b18af8e39150a88029ff"
server_key_version: "encryptionkey@0"
I20260924 10:07:19.438287 6268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "e7fc2d2cadc840d9b4a877f2ad84c6a9"
format_stamp: "Formatted at 2026-09-24 10:07:19 on dist-test-slave-s3n3"
server_key: "3ebd2356eeb3b4d89e487dc3422f6057"
server_key_iv: "f76ce2ba0630b18af8e39150a88029ff"
server_key_version: "encryptionkey@0"
I20260924 10:07:19.443544 6268 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260924 10:07:19.447166 6283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:19.448820 6268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:19.448963 6268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "e7fc2d2cadc840d9b4a877f2ad84c6a9"
format_stamp: "Formatted at 2026-09-24 10:07:19 on dist-test-slave-s3n3"
server_key: "3ebd2356eeb3b4d89e487dc3422f6057"
server_key_iv: "f76ce2ba0630b18af8e39150a88029ff"
server_key_version: "encryptionkey@0"
I20260924 10:07:19.449139 6268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:19.461526 6268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:19.465641 6268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:19.465883 6268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:19.466776 6268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:19.468190 6268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:19.468261 6268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:19.468343 6268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:19.468385 6268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:19.502303 6268 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:38585
I20260924 10:07:19.502350 6395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:38585 every 8 connection(s)
I20260924 10:07:19.503638 6268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:07:19.508996 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 6268
I20260924 10:07:19.509294 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:07:19.509912 6155 external_mini_cluster.cc:1428] Setting key 1497097cc4999ef2b46257e968054a7d
Sep 24 10:07:19 dist-test-slave-s3n3 krb5kdc[6162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:19.526607 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:19.506238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:47945 (local address 127.6.2.254:44405)
0924 10:07:19.506697 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:19.506703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:19.507622 (+ 919us) server_negotiation.cc:407] Connection header received
0924 10:07:19.509355 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:19.509361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:19.509456 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:19.509578 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:19.511203 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:19.512045 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:19.514566 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:19.514988 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:19.515203 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:19.519224 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:19.519235 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:19.519243 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:19.519288 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:19.523024 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:19.524018 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:19.524026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:19.524032 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:19.524190 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:19.524640 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:19.524648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:19.524654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:19.525071 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:19.525234 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:19.525991 (+ 757us) server_negotiation.cc:299] Negotiation successful
0924 10:07:19.526217 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":215,"threads_started":1}
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
I20260924 10:07:19.527797 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:19.506707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:44405 (local address 127.6.2.193:47945)
0924 10:07:19.507449 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:19.507485 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:19.509054 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:19.509812 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:19.509820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:19.510379 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:19.510962 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:19.510988 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:19.512207 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:19.512212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:19.514394 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:19.514414 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:19.515327 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:19.515336 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:19.515698 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:19.516507 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:19.516532 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:19.518942 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:19.523363 (+ 4421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:19.523375 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:19.523401 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:19.523870 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:19.524321 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:19.524324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:19.524327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:19.524522 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:19.525208 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:19.525214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:19.525555 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 10:07:19.525883 (+ 328us) client_negotiation.cc:240] Negotiation successful
0924 10:07:19.526515 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":318,"threads_started":1}
I20260924 10:07:19.529721 6396 heartbeater.cc:347] Connected to a master server at 127.6.2.254:44405
I20260924 10:07:19.530066 6396 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
I20260924 10:07:19.530965 6396 heartbeater.cc:511] Master 127.6.2.254:44405 requested a full tablet report, sending...
I20260924 10:07:19.533134 6207 ts_manager.cc:194] Registered new tserver with Master: e7fc2d2cadc840d9b4a877f2ad84c6a9 (127.6.2.193:38585)
I20260924 10:07:19.534570 6207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:47945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:07:19.550859 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:44405
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:19.884176 6402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:19.884955 6402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:19.885192 6402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:19.892997 6402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:19.893181 6402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:19.893378 6402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:07:19.904263 6402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:44405
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:19.905645 6402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:19.907001 6402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:19.920591 6408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:19.920648 6410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:19.920675 6407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:19.922053 6402 server_base.cc:1034] running on GCE node
I20260924 10:07:19.922693 6402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:19.923825 6402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:19.925066 6402 hybrid_clock.cc:648] HybridClock initialized: now 1790244439924999 us; error 63 us; skew 500 ppm
I20260924 10:07:19.925423 6402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:19 dist-test-slave-s3n3 krb5kdc[6162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:19.929733 6402 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:07:19.931927 6402 webserver.cc:460] Webserver started at http://127.6.2.194:44359/ using document root <none> and password file <none>
I20260924 10:07:19.933024 6402 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:19.933236 6402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:19.933843 6402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:19.937142 6402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "50291d8d44034ec9b899f1f211902094"
format_stamp: "Formatted at 2026-09-24 10:07:19 on dist-test-slave-s3n3"
server_key: "e99d6da5271242b7f66484014fa05b8b"
server_key_iv: "e535286356c082081ed3c067aefa0c4c"
server_key_version: "encryptionkey@0"
I20260924 10:07:19.938365 6402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "50291d8d44034ec9b899f1f211902094"
format_stamp: "Formatted at 2026-09-24 10:07:19 on dist-test-slave-s3n3"
server_key: "e99d6da5271242b7f66484014fa05b8b"
server_key_iv: "e535286356c082081ed3c067aefa0c4c"
server_key_version: "encryptionkey@0"
I20260924 10:07:19.945413 6402 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260924 10:07:19.950610 6417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:19.952323 6402 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:07:19.952504 6402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "50291d8d44034ec9b899f1f211902094"
format_stamp: "Formatted at 2026-09-24 10:07:19 on dist-test-slave-s3n3"
server_key: "e99d6da5271242b7f66484014fa05b8b"
server_key_iv: "e535286356c082081ed3c067aefa0c4c"
server_key_version: "encryptionkey@0"
I20260924 10:07:19.952713 6402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:19.973704 6402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:19.978277 6402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:19.978579 6402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:19.979606 6402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:19.980934 6402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:19.981009 6402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:19.981135 6402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:19.981196 6402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:20.013684 6402 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:43345
I20260924 10:07:20.013734 6529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:43345 every 8 connection(s)
I20260924 10:07:20.014961 6402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:07:20.019695 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 6402
I20260924 10:07:20.019899 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:07:20.020360 6155 external_mini_cluster.cc:1428] Setting key c3b7478f0d38689ddc4eae2b658a71a1
Sep 24 10:07:20 dist-test-slave-s3n3 krb5kdc[6162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:07:20.037695 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:20.017685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:47627 (local address 127.6.2.254:44405)
0924 10:07:20.017865 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:20.017870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:20.018992 (+ 1122us) server_negotiation.cc:407] Connection header received
0924 10:07:20.020983 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:20.020988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:20.021291 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:20.021448 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:20.023030 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.024293 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.025878 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.026590 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.026732 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:20.030700 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:20.030706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:20.030709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:20.030741 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:20.034121 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.035175 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.035179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.035181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.035325 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.035808 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.035814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.035818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.036172 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:20.036320 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:20.037067 (+ 747us) server_negotiation.cc:299] Negotiation successful
0924 10:07:20.037326 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:07:20.038766 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:20.018122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:44405 (local address 127.6.2.194:47627)
0924 10:07:20.018828 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:20.018865 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:20.020715 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:20.021675 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:20.021684 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:20.022298 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:20.022860 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:20.022876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.024491 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.024496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:20.025733 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:20.025745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.026825 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.026832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:20.027052 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:20.027837 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:20.027864 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:20.030465 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:20.034300 (+ 3835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:20.034311 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:20.034331 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:20.034975 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:20.035440 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:20.035447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:20.035450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:20.035702 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:20.036305 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:20.036314 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:20.036684 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 10:07:20.036954 (+ 270us) client_negotiation.cc:240] Negotiation successful
0924 10:07:20.037546 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":286,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
I20260924 10:07:20.040428 6530 heartbeater.cc:347] Connected to a master server at 127.6.2.254:44405
I20260924 10:07:20.040745 6530 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:20.041450 6530 heartbeater.cc:511] Master 127.6.2.254:44405 requested a full tablet report, sending...
I20260924 10:07:20.043033 6207 ts_manager.cc:194] Registered new tserver with Master: 50291d8d44034ec9b899f1f211902094 (127.6.2.194:43345)
I20260924 10:07:20.044137 6207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:47627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:07:20.063383 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:44405
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:20.394874 6535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:20.395249 6535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:20.395398 6535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:20.403106 6535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:20.403257 6535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:20.403430 6535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:07:20.413414 6535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:44405
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:20.414808 6535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:20.416200 6535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:20.429082 6541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:20.429059 6540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:20.429123 6543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:20.430269 6535 server_base.cc:1034] running on GCE node
I20260924 10:07:20.430903 6535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:20.431872 6535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:20.433104 6535 hybrid_clock.cc:648] HybridClock initialized: now 1790244440433054 us; error 64 us; skew 500 ppm
I20260924 10:07:20.433437 6535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:20 dist-test-slave-s3n3 krb5kdc[6162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:20.436645 6535 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:07:20.438077 6535 webserver.cc:460] Webserver started at http://127.6.2.195:43019/ using document root <none> and password file <none>
I20260924 10:07:20.438757 6535 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:20.438858 6535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:20.439224 6535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:20.442030 6535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "8d4c156384ec410c91a1e186047aef97"
format_stamp: "Formatted at 2026-09-24 10:07:20 on dist-test-slave-s3n3"
server_key: "527d2ecd0eb05dfbd523df546866bdcb"
server_key_iv: "7f747a116a8c3c5fca0752f7a01036cb"
server_key_version: "encryptionkey@0"
I20260924 10:07:20.442739 6535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "8d4c156384ec410c91a1e186047aef97"
format_stamp: "Formatted at 2026-09-24 10:07:20 on dist-test-slave-s3n3"
server_key: "527d2ecd0eb05dfbd523df546866bdcb"
server_key_iv: "7f747a116a8c3c5fca0752f7a01036cb"
server_key_version: "encryptionkey@0"
I20260924 10:07:20.448155 6535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 10:07:20.451977 6550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:20.453406 6535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:20.453572 6535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "8d4c156384ec410c91a1e186047aef97"
format_stamp: "Formatted at 2026-09-24 10:07:20 on dist-test-slave-s3n3"
server_key: "527d2ecd0eb05dfbd523df546866bdcb"
server_key_iv: "7f747a116a8c3c5fca0752f7a01036cb"
server_key_version: "encryptionkey@0"
I20260924 10:07:20.453773 6535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:20.482923 6535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:20.488696 6535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:20.489048 6535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:20.490413 6535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:20.492008 6535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:20.492136 6535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:20.492302 6535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:20.492386 6535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:20.526331 6535 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:32859
I20260924 10:07:20.526376 6662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:32859 every 8 connection(s)
I20260924 10:07:20.527721 6535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:07:20.531232 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 6535
I20260924 10:07:20.531442 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:07:20.531935 6155 external_mini_cluster.cc:1428] Setting key 785704e7249a77d1ff09f57e424c97e1
I20260924 10:07:20.538107 6396 heartbeater.cc:503] Master 127.6.2.254:44405 was elected leader, sending a full tablet report...
Sep 24 10:07:20 dist-test-slave-s3n3 krb5kdc[6162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:20.548882 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:20.530316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:52875 (local address 127.6.2.254:44405)
0924 10:07:20.530492 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:20.530497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:20.531786 (+ 1289us) server_negotiation.cc:407] Connection header received
0924 10:07:20.533837 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:20.533843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:20.533935 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:20.534055 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:20.536265 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.537012 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.538859 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.539333 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.539463 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:20.542832 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:20.542837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:20.542840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:20.542870 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:20.545462 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.546177 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.546181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.546184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.546362 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.546856 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.546860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.546863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.547251 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:20.547388 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:20.548315 (+ 927us) server_negotiation.cc:299] Negotiation successful
0924 10:07:20.548536 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:07:20.549903 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:20.530785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:44405 (local address 127.6.2.195:52875)
0924 10:07:20.531611 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:20.531661 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:20.533567 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:20.534450 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:20.534459 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:20.535000 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:20.536050 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:20.536072 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.537190 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.537194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:20.538688 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:20.538700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.539590 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.539598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:20.539864 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:20.540518 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:20.540541 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:20.542643 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:20.545607 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:20.545614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:20.545627 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:20.546045 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:20.546471 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:20.546474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:20.546485 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:20.546736 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:20.547383 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:20.547390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:20.547831 (+ 441us) client_negotiation.cc:769] Sending connection context
0924 10:07:20.548194 (+ 363us) client_negotiation.cc:240] Negotiation successful
0924 10:07:20.548711 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":360,"threads_started":1}
I20260924 10:07:20.551220 6663 heartbeater.cc:347] Connected to a master server at 127.6.2.254:44405
I20260924 10:07:20.551513 6663 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:20.552258 6663 heartbeater.cc:511] Master 127.6.2.254:44405 requested a full tablet report, sending...
I20260924 10:07:20.553659 6207 ts_manager.cc:194] Registered new tserver with Master: 8d4c156384ec410c91a1e186047aef97 (127.6.2.195:32859)
I20260924 10:07:20.554495 6207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:52875
I20260924 10:07:20.560930 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:07:20 dist-test-slave-s3n3 krb5kdc[6162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244440, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:07:20 dist-test-slave-s3n3 krb5kdc[6162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244440, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:07:20.607446 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:20.590700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.6.2.193:38585)
0924 10:07:20.591747 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:20.591759 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:20.591795 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:07:20.592044 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:20.592051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:20.592295 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:20.592489 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:20.593987 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.594982 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.596363 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.596957 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.597110 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:20.601233 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:20.601245 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:20.601260 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:20.601321 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:20.604556 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.605273 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.605281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.605284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.605409 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.605832 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.605842 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.605844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.606181 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:20.606505 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:20.606787 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 10:07:20.607053 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":488,"threads_started":1}
W20260924 10:07:20.608740 6311 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48956
Sep 24 10:07:20 dist-test-slave-s3n3 krb5kdc[6162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:07:20 dist-test-slave-s3n3 krb5kdc[6162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:07:20.657352 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:20.643093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.6.2.193:38585)
0924 10:07:20.643335 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:20.643341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:20.643525 (+ 184us) server_negotiation.cc:407] Connection header received
0924 10:07:20.643679 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:20.643683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:20.643753 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:20.643841 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:20.645176 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.645862 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.647244 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.647706 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.647815 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:20.651892 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:20.651897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:20.651901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:20.651936 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:20.654688 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.655439 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.655444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.655446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.655565 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.655964 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.655969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.655974 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.656220 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:20.656465 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:20.656773 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 10:07:20.657008 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 10:07:20.659006 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 6268
I20260924 10:07:20.672251 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:20.660100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.6.2.193:38585)
0924 10:07:20.660324 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:20.660329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:20.660534 (+ 205us) server_negotiation.cc:407] Connection header received
0924 10:07:20.660614 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:20.660617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:20.660680 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:20.660762 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:20.661951 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.662674 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.663938 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.664345 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.664440 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:20.666632 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:20.666638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:20.666641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:20.666671 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:20.669202 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.670254 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.670258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.670262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.670388 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.670835 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.670838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.670840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.671168 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:20.671364 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:20.671695 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 10:07:20.671860 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 10:07:20.673019 6391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:20.758677 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 6268
I20260924 10:07:20.776067 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 6402
Sep 24 10:07:20 dist-test-slave-s3n3 krb5kdc[6162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:07:20.793084 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:20.777036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.6.2.194:43345)
0924 10:07:20.777573 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:20.777580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:20.777607 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:07:20.777668 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:20.777675 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:20.777884 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:20.778052 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:20.779220 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.780232 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.781506 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.782077 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.782225 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:20.786715 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:20.786729 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:20.786750 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:20.786797 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:20.789788 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.790561 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.790569 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.790571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.790701 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.791514 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.791518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.791521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.791891 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:20.792169 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:20.792488 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 10:07:20.792691 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":277,"threads_started":1}
I20260924 10:07:20.794710 6525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:20.879961 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 6402
I20260924 10:07:20.900688 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 6535
Sep 24 10:07:20 dist-test-slave-s3n3 krb5kdc[6162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:07:20.917658 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:20.901797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36436 (local address 127.6.2.195:32859)
0924 10:07:20.902513 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:20.902520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:20.902552 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:07:20.902612 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:20.902618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:20.902835 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:20.903012 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:20.904297 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.905541 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.906921 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:20.907584 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:20.907729 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:20.911903 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:20.911915 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:20.911930 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:20.911965 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:20.914644 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.915495 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.915510 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.915512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.915630 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:20.916009 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:20.916019 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:20.916021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:20.916352 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:20.916678 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:20.917025 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 10:07:20.917308 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":388,"threads_started":1}
I20260924 10:07:20.918944 6658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:21.003880 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 6535
I20260924 10:07:21.019860 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 6175
I20260924 10:07:21.021164 6237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:21.123974 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 6175
2026-09-24T10:07:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4038 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:07:21 dist-test-slave-s3n3 krb5kdc[6684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:07:21 dist-test-slave-s3n3 krb5kdc[6684](info): set up 2 sockets
Sep 24 10:07:21 dist-test-slave-s3n3 krb5kdc[6684](info): commencing operation
krb5kdc: starting...
W20260924 10:07:22.236143 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:07:22 dist-test-slave-s3n3 krb5kdc[6684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:07:22.419548 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34487
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:35125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:22.739044 6697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:22.739429 6697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:22.739603 6697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:22.747035 6697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:22.747222 6697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:22.747306 6697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:22.747409 6697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:22.747470 6697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:22.758848 6697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:35125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:34487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:22.760329 6697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:22.761654 6697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:22.774145 6703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:22.774245 6702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:22.774245 6705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:22.775190 6697 server_base.cc:1034] running on GCE node
I20260924 10:07:22.776060 6697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:22.777597 6697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:22.778847 6697 hybrid_clock.cc:648] HybridClock initialized: now 1790244442778810 us; error 63 us; skew 500 ppm
I20260924 10:07:22.779289 6697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:22 dist-test-slave-s3n3 krb5kdc[6684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244442, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:22.787513 6697 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:07:22.789472 6697 webserver.cc:460] Webserver started at http://127.6.2.254:45695/ using document root <none> and password file <none>
I20260924 10:07:22.790287 6697 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:22.790426 6697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:22.790752 6697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:22.793830 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "d5bf24799d1f4ae899c154b6f60b02b5"
format_stamp: "Formatted at 2026-09-24 10:07:22 on dist-test-slave-s3n3"
server_key: "b44b54a3a4ce84ec2ad14de41c36f4d9"
server_key_iv: "1c7826aec6c5382de0b519ec32dadb93"
server_key_version: "encryptionkey@0"
I20260924 10:07:22.794606 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "d5bf24799d1f4ae899c154b6f60b02b5"
format_stamp: "Formatted at 2026-09-24 10:07:22 on dist-test-slave-s3n3"
server_key: "b44b54a3a4ce84ec2ad14de41c36f4d9"
server_key_iv: "1c7826aec6c5382de0b519ec32dadb93"
server_key_version: "encryptionkey@0"
I20260924 10:07:22.800680 6697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 10:07:22.805259 6712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:22.806802 6697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:07:22.807003 6697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "d5bf24799d1f4ae899c154b6f60b02b5"
format_stamp: "Formatted at 2026-09-24 10:07:22 on dist-test-slave-s3n3"
server_key: "b44b54a3a4ce84ec2ad14de41c36f4d9"
server_key_iv: "1c7826aec6c5382de0b519ec32dadb93"
server_key_version: "encryptionkey@0"
I20260924 10:07:22.807333 6697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:22.846072 6697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:22.850943 6697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:22.851277 6697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:22.870582 6697 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:34487
I20260924 10:07:22.870643 6763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:34487 every 8 connection(s)
I20260924 10:07:22.872056 6697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:22.875741 6764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:22.877286 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 6697
I20260924 10:07:22.877524 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:07:22.878212 6155 external_mini_cluster.cc:1428] Setting key 9e617e898ee4aec600fb67ce361cdef3
I20260924 10:07:22.888778 6764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5: Bootstrap starting.
I20260924 10:07:22.892241 6764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5: Neither blocks nor log segments found. Creating new log.
Sep 24 10:07:22 dist-test-slave-s3n3 krb5kdc[6684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:22.893272 6764 log.cc:824] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:22.896780 6764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5: No bootstrap required, opened a new log
I20260924 10:07:22.902730 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:22.880256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34486 (local address 127.6.2.254:34487)
0924 10:07:22.881155 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:22.881170 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:22.881221 (+ 51us) server_negotiation.cc:407] Connection header received
0924 10:07:22.883579 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:22.883606 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:22.884104 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:22.884525 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:22.885805 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:22.887833 (+ 2028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:22.889092 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:22.890058 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:22.890245 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:22.894293 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:22.894303 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:22.894318 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:22.894359 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:22.898270 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:22.898923 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:22.898931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:22.898939 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:22.899168 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:22.899542 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:22.899551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:22.899555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:22.900250 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:22.900525 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:22.900954 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 10:07:22.901393 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":361,"threads_started":1}
I20260924 10:07:22.906656 6764 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bf24799d1f4ae899c154b6f60b02b5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34487 } }
I20260924 10:07:22.906981 6764 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:22.907054 6764 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5bf24799d1f4ae899c154b6f60b02b5, State: Initialized, Role: FOLLOWER
I20260924 10:07:22.907730 6764 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bf24799d1f4ae899c154b6f60b02b5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34487 } }
I20260924 10:07:22.907933 6764 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:22.908027 6764 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:22.908146 6764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:22.910147 6764 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bf24799d1f4ae899c154b6f60b02b5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34487 } }
I20260924 10:07:22.910600 6764 leader_election.cc:302] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5bf24799d1f4ae899c154b6f60b02b5; no voters:
I20260924 10:07:22.911237 6764 leader_election.cc:288] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:22.911370 6769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:22.912063 6769 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [term 1 LEADER]: Becoming Leader. State: Replica: d5bf24799d1f4ae899c154b6f60b02b5, State: Running, Role: LEADER
I20260924 10:07:22.912595 6769 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bf24799d1f4ae899c154b6f60b02b5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34487 } }
I20260924 10:07:22.914418 6764 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:22.915318 6770 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5bf24799d1f4ae899c154b6f60b02b5. Latest consensus state: current_term: 1 leader_uuid: "d5bf24799d1f4ae899c154b6f60b02b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bf24799d1f4ae899c154b6f60b02b5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34487 } } }
I20260924 10:07:22.915743 6770 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:22.915335 6771 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5bf24799d1f4ae899c154b6f60b02b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5bf24799d1f4ae899c154b6f60b02b5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34487 } } }
I20260924 10:07:22.916143 6771 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:22.917661 6774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:22.924094 6774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:22.935120 6774 catalog_manager.cc:1347] Generated new cluster ID: 8fd850626c94487898b16c9897fa7d21
I20260924 10:07:22.935242 6774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:22.953279 6774 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:22.955133 6774 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:22.968197 6774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5bf24799d1f4ae899c154b6f60b02b5: Generated new TSK 0
I20260924 10:07:22.969137 6774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:07:23.015002 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34487
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:35125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:23.370035 6790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:23.370319 6790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:23.370482 6790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:23.378012 6790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:23.378139 6790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:23.378317 6790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:07:23.388458 6790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:35125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34487
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:23.389804 6790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:23.391201 6790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:23.404726 6796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:23.404966 6795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:23.404701 6798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:23.405537 6790 server_base.cc:1034] running on GCE node
I20260924 10:07:23.406215 6790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:23.407191 6790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:23.408440 6790 hybrid_clock.cc:648] HybridClock initialized: now 1790244443408388 us; error 41 us; skew 500 ppm
I20260924 10:07:23.408783 6790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:23 dist-test-slave-s3n3 krb5kdc[6684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244443, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:23.412576 6790 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:07:23.414149 6790 webserver.cc:460] Webserver started at http://127.6.2.193:39203/ using document root <none> and password file <none>
I20260924 10:07:23.414878 6790 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:23.415002 6790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:23.415339 6790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:23.418228 6790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "53b48d30532b4e1c863fabf3f766df0c"
format_stamp: "Formatted at 2026-09-24 10:07:23 on dist-test-slave-s3n3"
server_key: "e8e9d8239579b47aeacc8eb98741dc76"
server_key_iv: "5240ba7c409b7b30bf14b3bd107dba16"
server_key_version: "encryptionkey@0"
I20260924 10:07:23.419021 6790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "53b48d30532b4e1c863fabf3f766df0c"
format_stamp: "Formatted at 2026-09-24 10:07:23 on dist-test-slave-s3n3"
server_key: "e8e9d8239579b47aeacc8eb98741dc76"
server_key_iv: "5240ba7c409b7b30bf14b3bd107dba16"
server_key_version: "encryptionkey@0"
I20260924 10:07:23.424058 6790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:07:23.427479 6805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:23.429015 6790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:07:23.429217 6790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "53b48d30532b4e1c863fabf3f766df0c"
format_stamp: "Formatted at 2026-09-24 10:07:23 on dist-test-slave-s3n3"
server_key: "e8e9d8239579b47aeacc8eb98741dc76"
server_key_iv: "5240ba7c409b7b30bf14b3bd107dba16"
server_key_version: "encryptionkey@0"
I20260924 10:07:23.429406 6790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:23.446605 6790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:23.452417 6790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:23.452817 6790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:23.454105 6790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:23.455662 6790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:23.455741 6790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:23.455891 6790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:23.455971 6790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:23.489192 6790 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:44191
I20260924 10:07:23.489223 6917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:44191 every 8 connection(s)
I20260924 10:07:23.490495 6790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:07:23.495221 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 6790
I20260924 10:07:23.495425 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:07:23.496031 6155 external_mini_cluster.cc:1428] Setting key c2c3f209bf539e50c0e6a493ad6bf65c
Sep 24 10:07:23 dist-test-slave-s3n3 krb5kdc[6684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244443, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
I20260924 10:07:23.511927 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:23.493164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:43509 (local address 127.6.2.254:34487)
0924 10:07:23.493625 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:23.493630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:23.494654 (+ 1024us) server_negotiation.cc:407] Connection header received
0924 10:07:23.496737 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:23.496764 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:23.496849 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:23.496985 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:23.498787 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:23.499567 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:23.501094 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:23.501540 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:23.501702 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:23.505738 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:23.505743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:23.505746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:23.505777 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:23.508370 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:23.509110 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:23.509117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:23.509120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:23.509243 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:23.509731 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:23.509737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:23.509744 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:23.510077 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:23.510227 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:23.511197 (+ 970us) server_negotiation.cc:299] Negotiation successful
0924 10:07:23.511436 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":225,"threads_started":1}
I20260924 10:07:23.512976 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:23.493698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34487 (local address 127.6.2.193:43509)
0924 10:07:23.494462 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:23.494502 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:23.496413 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:23.497280 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:23.497290 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:23.498043 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:23.498617 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:23.498636 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:23.499741 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:23.499745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:23.500946 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:23.500958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:23.501758 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:23.501763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:23.501965 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:23.502726 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:23.502760 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:23.505488 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:23.508522 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:23.508532 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:23.508549 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:23.508978 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:23.509354 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:23.509359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:23.509362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:23.509613 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:23.510202 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:23.510211 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:23.510735 (+ 524us) client_negotiation.cc:769] Sending connection context
0924 10:07:23.511088 (+ 353us) client_negotiation.cc:240] Negotiation successful
0924 10:07:23.511651 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":294,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
I20260924 10:07:23.514916 6918 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34487
I20260924 10:07:23.515272 6918 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:23.516168 6918 heartbeater.cc:511] Master 127.6.2.254:34487 requested a full tablet report, sending...
I20260924 10:07:23.518522 6729 ts_manager.cc:194] Registered new tserver with Master: 53b48d30532b4e1c863fabf3f766df0c (127.6.2.193:44191)
I20260924 10:07:23.520490 6729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:43509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:07:23.538569 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34487
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:35125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:23.866489 6924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:23.866777 6924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:23.866945 6924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:23.874382 6924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:23.874492 6924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:23.874732 6924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:07:23.885022 6924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:35125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34487
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:23.886332 6924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:23.887678 6924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:23.900662 6932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:23.900665 6929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:23.900696 6930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:23.901857 6924 server_base.cc:1034] running on GCE node
I20260924 10:07:23.902521 6924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:23.903585 6924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:23.904829 6924 hybrid_clock.cc:648] HybridClock initialized: now 1790244443904805 us; error 54 us; skew 500 ppm
I20260924 10:07:23.905184 6924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:23 dist-test-slave-s3n3 krb5kdc[6684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244443, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:23.908466 6924 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:07:23.910025 6924 webserver.cc:460] Webserver started at http://127.6.2.194:38767/ using document root <none> and password file <none>
I20260924 10:07:23.910872 6924 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:23.911015 6924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:23.911362 6924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:23.914247 6924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "44459dc2fca3488094ad8d6cef866da5"
format_stamp: "Formatted at 2026-09-24 10:07:23 on dist-test-slave-s3n3"
server_key: "014094a83c1fac90046f9d348ead1b97"
server_key_iv: "21f97fc5b7fd57d8068bbb2d94435d63"
server_key_version: "encryptionkey@0"
I20260924 10:07:23.915113 6924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "44459dc2fca3488094ad8d6cef866da5"
format_stamp: "Formatted at 2026-09-24 10:07:23 on dist-test-slave-s3n3"
server_key: "014094a83c1fac90046f9d348ead1b97"
server_key_iv: "21f97fc5b7fd57d8068bbb2d94435d63"
server_key_version: "encryptionkey@0"
I20260924 10:07:23.920315 6924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:07:23.923936 6939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:23.925323 6924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:23.925495 6924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "44459dc2fca3488094ad8d6cef866da5"
format_stamp: "Formatted at 2026-09-24 10:07:23 on dist-test-slave-s3n3"
server_key: "014094a83c1fac90046f9d348ead1b97"
server_key_iv: "21f97fc5b7fd57d8068bbb2d94435d63"
server_key_version: "encryptionkey@0"
I20260924 10:07:23.925694 6924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:23.937786 6924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:23.942204 6924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:23.942505 6924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:23.943794 6924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:23.945163 6924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:23.945267 6924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:23.945416 6924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:23.945487 6924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:23.980276 6924 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:43953
I20260924 10:07:23.980329 7051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:43953 every 8 connection(s)
I20260924 10:07:23.981576 6924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:07:23.986690 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 6924
I20260924 10:07:23.986882 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:07:23.987466 6155 external_mini_cluster.cc:1428] Setting key 2b6abe82163586ba2e45b71ea48731bd
Sep 24 10:07:23 dist-test-slave-s3n3 krb5kdc[6684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244443, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:07:24.004029 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:23.984307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:42861 (local address 127.6.2.254:34487)
0924 10:07:23.984478 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:23.984483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:23.985537 (+ 1054us) server_negotiation.cc:407] Connection header received
0924 10:07:23.987991 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:23.987996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:23.988064 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:23.988185 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:23.989681 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:23.990385 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:23.992828 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:23.993506 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:23.993699 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:23.997695 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:23.997700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:23.997704 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:23.997740 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:24.000404 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.001448 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.001452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.001455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.001574 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.002105 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.002108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.002110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.002397 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:24.002504 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:24.003586 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0924 10:07:24.003781 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:07:24.005277 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:23.984660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34487 (local address 127.6.2.194:42861)
0924 10:07:23.985366 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:23.985403 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:23.987684 (+ 2281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:23.988427 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:23.988435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:23.988952 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:23.989519 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:23.989536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:23.990526 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:23.990531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:23.992628 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:23.992643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:23.993812 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:23.993820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:23.994064 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:23.994812 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:23.994835 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:23.997438 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:24.000567 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:24.000580 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:24.000603 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:24.001301 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:24.001691 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:24.001698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:24.001703 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:24.001971 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:24.002546 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:24.002558 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:24.003096 (+ 538us) client_negotiation.cc:769] Sending connection context
0924 10:07:24.003434 (+ 338us) client_negotiation.cc:240] Negotiation successful
0924 10:07:24.003980 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":285,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
I20260924 10:07:24.006886 7052 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34487
I20260924 10:07:24.007274 7052 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:24.007995 7052 heartbeater.cc:511] Master 127.6.2.254:34487 requested a full tablet report, sending...
I20260924 10:07:24.009507 6729 ts_manager.cc:194] Registered new tserver with Master: 44459dc2fca3488094ad8d6cef866da5 (127.6.2.194:43953)
I20260924 10:07:24.010345 6729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:42861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:07:24.030640 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34487
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:35125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:24.357313 7057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:24.357601 7057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:24.357764 7057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:24.365224 7057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:24.365340 7057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:24.365550 7057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:07:24.375543 7057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:35125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34487
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:24.376864 7057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:24.378211 7057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:24.391870 7062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:24.391836 7063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:24.391785 7065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:24.392937 7057 server_base.cc:1034] running on GCE node
I20260924 10:07:24.393585 7057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:24.394598 7057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:24.395823 7057 hybrid_clock.cc:648] HybridClock initialized: now 1790244444395758 us; error 63 us; skew 500 ppm
I20260924 10:07:24.396178 7057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:24 dist-test-slave-s3n3 krb5kdc[6684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244444, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:24.399595 7057 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:07:24.401224 7057 webserver.cc:460] Webserver started at http://127.6.2.195:41783/ using document root <none> and password file <none>
I20260924 10:07:24.401937 7057 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:24.402117 7057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:24.402438 7057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:24.405411 7057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "40eaf3b5891f4eefa1a2a3d82f48c145"
format_stamp: "Formatted at 2026-09-24 10:07:24 on dist-test-slave-s3n3"
server_key: "80ce9c99df620e898435c60bc0eda407"
server_key_iv: "8b1d984b2e337fced8ccfcac72b95031"
server_key_version: "encryptionkey@0"
I20260924 10:07:24.406214 7057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "40eaf3b5891f4eefa1a2a3d82f48c145"
format_stamp: "Formatted at 2026-09-24 10:07:24 on dist-test-slave-s3n3"
server_key: "80ce9c99df620e898435c60bc0eda407"
server_key_iv: "8b1d984b2e337fced8ccfcac72b95031"
server_key_version: "encryptionkey@0"
I20260924 10:07:24.411477 7057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 10:07:24.414991 7072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:24.416465 7057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:24.416651 7057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "40eaf3b5891f4eefa1a2a3d82f48c145"
format_stamp: "Formatted at 2026-09-24 10:07:24 on dist-test-slave-s3n3"
server_key: "80ce9c99df620e898435c60bc0eda407"
server_key_iv: "8b1d984b2e337fced8ccfcac72b95031"
server_key_version: "encryptionkey@0"
I20260924 10:07:24.416858 7057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:24.449838 7057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:24.453986 7057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:24.454313 7057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:24.455413 7057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:24.456710 7057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:24.456812 7057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:24.456969 7057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:24.457060 7057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:24.491137 7057 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:45889
I20260924 10:07:24.491170 7184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:45889 every 8 connection(s)
I20260924 10:07:24.492408 7057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:07:24.498719 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 7057
I20260924 10:07:24.498924 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:07:24.499437 6155 external_mini_cluster.cc:1428] Setting key aae4b6b3f54824a3ae1fec21eac78e2d
Sep 24 10:07:24 dist-test-slave-s3n3 krb5kdc[6684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244444, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:24.512363 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.495038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:56927 (local address 127.6.2.254:34487)
0924 10:07:24.495246 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:24.495251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:24.496402 (+ 1151us) server_negotiation.cc:407] Connection header received
0924 10:07:24.498190 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:24.498195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:24.498285 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:24.498399 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:24.500104 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.501306 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.503006 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.503492 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.503592 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.506871 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:24.506876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:24.506879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:24.506910 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:24.509396 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.510054 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.510058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.510060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.510208 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.510583 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.510587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.510589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.510913 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:24.511031 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:24.511755 (+ 724us) server_negotiation.cc:299] Negotiation successful
0924 10:07:24.512002 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 10:07:24.513450 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.495520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34487 (local address 127.6.2.195:56927)
0924 10:07:24.496240 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:24.496277 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:24.497923 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:24.498633 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:24.498641 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:24.499294 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:24.499899 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:24.499941 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.501452 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.501456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:24.502877 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:24.502889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.503694 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.503700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:24.503892 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.504609 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:24.504635 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:24.506695 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:24.509519 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:24.509526 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:24.509539 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:24.509955 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:24.510299 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:24.510302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:24.510304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:24.510463 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:24.511093 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:24.511101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:24.511435 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 10:07:24.511665 (+ 230us) client_negotiation.cc:240] Negotiation successful
0924 10:07:24.512275 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":327,"threads_started":1}
I20260924 10:07:24.514937 7185 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34487
I20260924 10:07:24.515321 7185 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:24.516021 7185 heartbeater.cc:511] Master 127.6.2.254:34487 requested a full tablet report, sending...
I20260924 10:07:24.517387 6728 ts_manager.cc:194] Registered new tserver with Master: 40eaf3b5891f4eefa1a2a3d82f48c145 (127.6.2.195:45889)
I20260924 10:07:24.518247 6728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:56927
I20260924 10:07:24.523491 6918 heartbeater.cc:503] Master 127.6.2.254:34487 was elected leader, sending a full tablet report...
I20260924 10:07:24.526988 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:07:24.546325 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.533537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.6.2.254:34487)
0924 10:07:24.533765 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:24.533769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:24.534010 (+ 241us) server_negotiation.cc:407] Connection header received
0924 10:07:24.534135 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:24.534138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:24.534255 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:24.534392 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:24.535897 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.536613 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.537928 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.538352 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.538494 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.540886 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:24.540894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:24.540902 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:24.540932 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:24.543508 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.544227 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.544241 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.544247 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.544406 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.544834 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.544845 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.544848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.545188 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:24.545450 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:24.545795 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 10:07:24.546006 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 10:07:24.553002 6728 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:07:24.559875 6728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:07:24.587302 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.572997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.6.2.193:44191)
0924 10:07:24.575204 (+ 2207us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:24.575219 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:24.575257 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:07:24.575386 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:24.575398 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:24.575727 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:24.576934 (+ 1207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:24.577989 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.580076 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.584044 (+ 3968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.585708 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.585842 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.586110 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:24.586706 (+ 596us) server_negotiation.cc:299] Negotiation successful
0924 10:07:24.586921 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1959,"thread_start_us":1788,"threads_started":1}
I20260924 10:07:24.587968 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.572614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:44191 (local address 127.0.0.1:37818)
0924 10:07:24.573854 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:24.573906 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:24.574139 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:24.576024 (+ 1885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:24.576037 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:24.576083 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:24.576665 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:24.576683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.581001 (+ 4318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.581010 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:24.583809 (+ 2799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:24.583830 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.586097 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.586105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:24.586409 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.586474 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 10:07:24.586630 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 10:07:24.586826 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":50,"thread_start_us":695,"threads_started":1}
I20260924 10:07:24.590194 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.572984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:45889 (local address 127.0.0.1:39704)
0924 10:07:24.580711 (+ 7727us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:24.580728 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:24.580834 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:24.582603 (+ 1769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:24.582607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:24.582632 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:24.583100 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:24.583113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.585342 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.585352 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:24.587465 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:24.587480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.589404 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.589410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:24.589579 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.589614 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:07:24.589731 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 10:07:24.589858 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7585,"thread_start_us":7436,"threads_started":1}
I20260924 10:07:24.590286 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.573687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.6.2.195:45889)
0924 10:07:24.581949 (+ 8262us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:24.581957 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:24.581991 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:07:24.582095 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:24.582106 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:24.582368 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:24.583319 (+ 951us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:24.583575 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.585175 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.587654 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.589198 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.589362 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.589598 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:24.589783 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 10:07:24.589938 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8096,"thread_start_us":7929,"threads_started":1}
I20260924 10:07:24.591584 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.574160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:43953 (local address 127.0.0.1:52278)
0924 10:07:24.582362 (+ 8202us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:24.582380 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:24.582502 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:24.583406 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:24.583410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:24.583432 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:24.583817 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:24.583825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.585167 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.585173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:24.588134 (+ 2961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:24.588148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.590721 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.590729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:24.590902 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.590934 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:07:24.591036 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 10:07:24.591204 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8084,"mutex_wait_us":99,"thread_start_us":7869,"threads_started":1}
I20260924 10:07:24.591809 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.574683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.6.2.194:43953)
0924 10:07:24.581412 (+ 6729us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:24.581422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:24.582637 (+ 1215us) server_negotiation.cc:407] Connection header received
0924 10:07:24.582738 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:24.582745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:24.582957 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:24.583176 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:24.583980 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.584949 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.588332 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.590178 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.590412 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.590695 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:24.591166 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 10:07:24.591389 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6560,"thread_start_us":6452,"threads_started":1}
I20260924 10:07:24.593974 6853 tablet_service.cc:1467] Processing CreateTablet for tablet df8b0ce16b964f1890c0d63d681f4e71 (DEFAULT_TABLE table=test-table [id=0a3bb65c39f14c5f9e5a1c65d963f66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:24.594344 7120 tablet_service.cc:1467] Processing CreateTablet for tablet df8b0ce16b964f1890c0d63d681f4e71 (DEFAULT_TABLE table=test-table [id=0a3bb65c39f14c5f9e5a1c65d963f66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:24.595640 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8b0ce16b964f1890c0d63d681f4e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:24.595708 6853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8b0ce16b964f1890c0d63d681f4e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:24.596189 6987 tablet_service.cc:1467] Processing CreateTablet for tablet df8b0ce16b964f1890c0d63d681f4e71 (DEFAULT_TABLE table=test-table [id=0a3bb65c39f14c5f9e5a1c65d963f66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:24.597451 6987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8b0ce16b964f1890c0d63d681f4e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:24.607187 7203 tablet_bootstrap.cc:492] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145: Bootstrap starting.
I20260924 10:07:24.610260 7203 tablet_bootstrap.cc:654] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:24.610838 7204 tablet_bootstrap.cc:492] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c: Bootstrap starting.
I20260924 10:07:24.611744 7203 log.cc:824] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:24.612712 7205 tablet_bootstrap.cc:492] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5: Bootstrap starting.
I20260924 10:07:24.614948 7204 tablet_bootstrap.cc:654] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:24.616197 7204 log.cc:824] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:24.616240 7205 tablet_bootstrap.cc:654] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:24.616348 7203 tablet_bootstrap.cc:492] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145: No bootstrap required, opened a new log
I20260924 10:07:24.616632 7203 ts_tablet_manager.cc:1397] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 10:07:24.617475 7205 log.cc:824] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:24.618999 7204 tablet_bootstrap.cc:492] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c: No bootstrap required, opened a new log
I20260924 10:07:24.619290 7204 ts_tablet_manager.cc:1397] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260924 10:07:24.620342 7205 tablet_bootstrap.cc:492] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5: No bootstrap required, opened a new log
I20260924 10:07:24.620584 7205 ts_tablet_manager.cc:1397] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 10:07:24.627031 7204 raft_consensus.cc:348] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b48d30532b4e1c863fabf3f766df0c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 44191 } } peers { permanent_uuid: "40eaf3b5891f4eefa1a2a3d82f48c145" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 45889 } } peers { permanent_uuid: "44459dc2fca3488094ad8d6cef866da5" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43953 } }
I20260924 10:07:24.627049 7203 raft_consensus.cc:348] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b48d30532b4e1c863fabf3f766df0c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 44191 } } peers { permanent_uuid: "40eaf3b5891f4eefa1a2a3d82f48c145" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 45889 } } peers { permanent_uuid: "44459dc2fca3488094ad8d6cef866da5" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43953 } }
I20260924 10:07:24.627482 7204 raft_consensus.cc:374] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:24.627498 7203 raft_consensus.cc:374] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:24.627590 7204 raft_consensus.cc:729] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b48d30532b4e1c863fabf3f766df0c, State: Initialized, Role: FOLLOWER
I20260924 10:07:24.627632 7203 raft_consensus.cc:729] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40eaf3b5891f4eefa1a2a3d82f48c145, State: Initialized, Role: FOLLOWER
I20260924 10:07:24.628296 7204 consensus_queue.cc:260] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b48d30532b4e1c863fabf3f766df0c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 44191 } } peers { permanent_uuid: "40eaf3b5891f4eefa1a2a3d82f48c145" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 45889 } } peers { permanent_uuid: "44459dc2fca3488094ad8d6cef866da5" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43953 } }
I20260924 10:07:24.628266 7203 consensus_queue.cc:260] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b48d30532b4e1c863fabf3f766df0c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 44191 } } peers { permanent_uuid: "40eaf3b5891f4eefa1a2a3d82f48c145" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 45889 } } peers { permanent_uuid: "44459dc2fca3488094ad8d6cef866da5" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43953 } }
I20260924 10:07:24.629624 7185 heartbeater.cc:503] Master 127.6.2.254:34487 was elected leader, sending a full tablet report...
I20260924 10:07:24.630527 7203 ts_tablet_manager.cc:1428] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260924 10:07:24.631214 7204 ts_tablet_manager.cc:1428] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260924 10:07:24.632817 7205 raft_consensus.cc:348] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b48d30532b4e1c863fabf3f766df0c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 44191 } } peers { permanent_uuid: "40eaf3b5891f4eefa1a2a3d82f48c145" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 45889 } } peers { permanent_uuid: "44459dc2fca3488094ad8d6cef866da5" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43953 } }
I20260924 10:07:24.633318 7205 raft_consensus.cc:374] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:24.633495 7205 raft_consensus.cc:729] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44459dc2fca3488094ad8d6cef866da5, State: Initialized, Role: FOLLOWER
I20260924 10:07:24.635159 7205 consensus_queue.cc:260] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b48d30532b4e1c863fabf3f766df0c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 44191 } } peers { permanent_uuid: "40eaf3b5891f4eefa1a2a3d82f48c145" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 45889 } } peers { permanent_uuid: "44459dc2fca3488094ad8d6cef866da5" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43953 } }
I20260924 10:07:24.636343 7052 heartbeater.cc:503] Master 127.6.2.254:34487 was elected leader, sending a full tablet report...
I20260924 10:07:24.636952 7205 ts_tablet_manager.cc:1428] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20260924 10:07:24.640700 7210 raft_consensus.cc:482] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:07:24.640974 7210 raft_consensus.cc:504] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b48d30532b4e1c863fabf3f766df0c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 44191 } } peers { permanent_uuid: "40eaf3b5891f4eefa1a2a3d82f48c145" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 45889 } } peers { permanent_uuid: "44459dc2fca3488094ad8d6cef866da5" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43953 } }
I20260924 10:07:24.642521 7210 leader_election.cc:288] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40eaf3b5891f4eefa1a2a3d82f48c145 (127.6.2.195:45889), 44459dc2fca3488094ad8d6cef866da5 (127.6.2.194:43953)
I20260924 10:07:24.649139 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.643248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:34525 (local address 127.6.2.194:43953)
0924 10:07:24.643621 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:24.643627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:24.643823 (+ 196us) server_negotiation.cc:407] Connection header received
0924 10:07:24.644013 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:24.644020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:24.644080 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:24.644174 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:24.644868 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.645552 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.647439 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.648204 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.648368 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.648467 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:24.648746 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 10:07:24.648871 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260924 10:07:24.649310 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.642885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:43953 (local address 127.6.2.193:34525)
0924 10:07:24.643740 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:24.643757 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:24.643880 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:24.644187 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:24.644191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:24.644213 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:24.644683 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:24.644692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.645696 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.645702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:24.647290 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:24.647302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.648419 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.648425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:24.648583 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.648606 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:07:24.648723 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 10:07:24.648858 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":7,"thread_start_us":427,"threads_started":1}
I20260924 10:07:24.649477 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.643364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:35469 (local address 127.6.2.195:45889)
0924 10:07:24.643729 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:24.643735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:24.643756 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:07:24.643965 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:24.643969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:24.644029 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:24.644126 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:24.644810 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.645609 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.647628 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.648460 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.648627 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.648743 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:24.649051 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 10:07:24.649183 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260924 10:07:24.649643 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.642868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:45889 (local address 127.6.2.193:35469)
0924 10:07:24.643619 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:24.643640 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:24.643812 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:24.644154 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:24.644158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:24.644184 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:24.644654 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:24.644667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.645757 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.645763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:24.647474 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:24.647485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.648708 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.648714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:24.648882 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.648906 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:07:24.649027 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 10:07:24.649165 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":70,"thread_start_us":772,"threads_started":1}
I20260924 10:07:24.650619 7007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8b0ce16b964f1890c0d63d681f4e71" candidate_uuid: "53b48d30532b4e1c863fabf3f766df0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44459dc2fca3488094ad8d6cef866da5" is_pre_election: true
I20260924 10:07:24.650941 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8b0ce16b964f1890c0d63d681f4e71" candidate_uuid: "53b48d30532b4e1c863fabf3f766df0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40eaf3b5891f4eefa1a2a3d82f48c145" is_pre_election: true
I20260924 10:07:24.651190 7007 raft_consensus.cc:2454] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53b48d30532b4e1c863fabf3f766df0c in term 0.
I20260924 10:07:24.651489 7140 raft_consensus.cc:2454] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53b48d30532b4e1c863fabf3f766df0c in term 0.
I20260924 10:07:24.651963 6806 leader_election.cc:302] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44459dc2fca3488094ad8d6cef866da5, 53b48d30532b4e1c863fabf3f766df0c; no voters:
I20260924 10:07:24.652369 7210 raft_consensus.cc:2781] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:07:24.652487 7210 raft_consensus.cc:482] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:07:24.652627 7210 raft_consensus.cc:3037] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:24.655221 7210 raft_consensus.cc:504] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b48d30532b4e1c863fabf3f766df0c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 44191 } } peers { permanent_uuid: "40eaf3b5891f4eefa1a2a3d82f48c145" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 45889 } } peers { permanent_uuid: "44459dc2fca3488094ad8d6cef866da5" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43953 } }
I20260924 10:07:24.655920 7210 leader_election.cc:288] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [CANDIDATE]: Term 1 election: Requested vote from peers 40eaf3b5891f4eefa1a2a3d82f48c145 (127.6.2.195:45889), 44459dc2fca3488094ad8d6cef866da5 (127.6.2.194:43953)
I20260924 10:07:24.656353 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8b0ce16b964f1890c0d63d681f4e71" candidate_uuid: "53b48d30532b4e1c863fabf3f766df0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40eaf3b5891f4eefa1a2a3d82f48c145"
I20260924 10:07:24.656383 7007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8b0ce16b964f1890c0d63d681f4e71" candidate_uuid: "53b48d30532b4e1c863fabf3f766df0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44459dc2fca3488094ad8d6cef866da5"
I20260924 10:07:24.656579 7007 raft_consensus.cc:3037] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:24.656595 7140 raft_consensus.cc:3037] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:24.658751 7007 raft_consensus.cc:2454] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53b48d30532b4e1c863fabf3f766df0c in term 1.
I20260924 10:07:24.658757 7140 raft_consensus.cc:2454] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53b48d30532b4e1c863fabf3f766df0c in term 1.
I20260924 10:07:24.659165 6809 leader_election.cc:302] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40eaf3b5891f4eefa1a2a3d82f48c145, 53b48d30532b4e1c863fabf3f766df0c; no voters:
I20260924 10:07:24.659488 7210 raft_consensus.cc:2781] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:24.659876 7210 raft_consensus.cc:686] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [term 1 LEADER]: Becoming Leader. State: Replica: 53b48d30532b4e1c863fabf3f766df0c, State: Running, Role: LEADER
I20260924 10:07:24.660351 7210 consensus_queue.cc:237] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b48d30532b4e1c863fabf3f766df0c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 44191 } } peers { permanent_uuid: "40eaf3b5891f4eefa1a2a3d82f48c145" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 45889 } } peers { permanent_uuid: "44459dc2fca3488094ad8d6cef866da5" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43953 } }
I20260924 10:07:24.665972 6729 catalog_manager.cc:5495] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 53b48d30532b4e1c863fabf3f766df0c (127.6.2.193). New cstate: current_term: 1 leader_uuid: "53b48d30532b4e1c863fabf3f766df0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b48d30532b4e1c863fabf3f766df0c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 44191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40eaf3b5891f4eefa1a2a3d82f48c145" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 45889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44459dc2fca3488094ad8d6cef866da5" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43953 } health_report { overall_health: UNKNOWN } } }
Sep 24 10:07:24 dist-test-slave-s3n3 krb5kdc[6684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:07:24.692528 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.677729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.6.2.193:44191)
0924 10:07:24.677900 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:24.677904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:24.678400 (+ 496us) server_negotiation.cc:407] Connection header received
0924 10:07:24.678533 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:24.678537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:24.678602 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:24.678700 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:24.680199 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.680906 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.682224 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.682625 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.682726 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.686785 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:24.686801 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:24.686818 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:24.686855 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:24.689807 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.690479 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.690486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.690489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.690653 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.690985 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.690989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.690991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.691438 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:24.691670 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:24.691952 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 10:07:24.692145 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 10:07:24 dist-test-slave-s3n3 krb5kdc[6684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244444, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260924 10:07:24.733772 7053 tablet.cc:2369] T df8b0ce16b964f1890c0d63d681f4e71 P 44459dc2fca3488094ad8d6cef866da5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:07:24 dist-test-slave-s3n3 krb5kdc[6684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244444, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:07:24.740552 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.725311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.6.2.193:44191)
0924 10:07:24.725633 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:24.725639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:24.725838 (+ 199us) server_negotiation.cc:407] Connection header received
0924 10:07:24.726020 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:24.726025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:24.726090 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:24.726225 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:24.727577 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.728451 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.730140 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.730653 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.730758 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.735274 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:24.735280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:24.735283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:24.735317 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:24.737793 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.738460 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.738472 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.738475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.738604 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.738991 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.738996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.739000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.739387 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:24.739714 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:24.739977 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 10:07:24.740216 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260924 10:07:24.742971 6919 tablet.cc:2369] T df8b0ce16b964f1890c0d63d681f4e71 P 53b48d30532b4e1c863fabf3f766df0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:07:24.744128 7186 tablet.cc:2369] T df8b0ce16b964f1890c0d63d681f4e71 P 40eaf3b5891f4eefa1a2a3d82f48c145: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:07:24 dist-test-slave-s3n3 krb5kdc[6684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:07:24 dist-test-slave-s3n3 krb5kdc[6684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:07:24.787297 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.772637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.6.2.193:44191)
0924 10:07:24.772881 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:24.772888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:24.773106 (+ 218us) server_negotiation.cc:407] Connection header received
0924 10:07:24.773337 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:24.773347 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:24.773423 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:24.773534 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:24.774879 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.775732 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.776942 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.777380 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.777517 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.781547 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:24.781552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:24.781555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:24.781590 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:24.784580 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.785301 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.785307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.785310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.785451 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.785835 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.785843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.785846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.786129 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:24.786402 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:24.786676 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 10:07:24.786900 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260924 10:07:24.788986 6833 tablet_service.cc:2908] Rejecting scan request for tablet df8b0ce16b964f1890c0d63d681f4e71: Uninitialized: clean time has not yet been initialized
I20260924 10:07:24.792510 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 6790
I20260924 10:07:24.805202 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.793380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.6.2.193:44191)
0924 10:07:24.793584 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:24.793593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:24.793927 (+ 334us) server_negotiation.cc:407] Connection header received
0924 10:07:24.794039 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:24.794042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:24.794115 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:24.794194 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:24.795345 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.796087 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.797282 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.797694 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.797807 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.800019 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:24.800024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:24.800027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:24.800056 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:24.802639 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.803317 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.803321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.803324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.803435 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.803777 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.803781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.803783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.804031 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:24.804227 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:24.804646 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 10:07:24.804907 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:07:24.806244 6913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:24.900593 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 6790
I20260924 10:07:24.917979 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 6924
Sep 24 10:07:24 dist-test-slave-s3n3 krb5kdc[6684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:07:24.933692 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:24.918799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.6.2.194:43953)
0924 10:07:24.918982 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:24.918988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:24.919009 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:07:24.919126 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:24.919136 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:24.919232 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:24.919345 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:24.920636 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.921488 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.922760 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:24.923208 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:24.923344 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:24.927370 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:24.927383 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:24.927421 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:24.927479 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:24.930713 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.931596 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.931603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.931605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.931732 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:24.932133 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:24.932138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:24.932141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:24.932530 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:24.932870 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:24.933111 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 10:07:24.933344 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:07:24.934518 7047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:25.024343 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 6924
I20260924 10:07:25.040885 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 7057
Sep 24 10:07:25 dist-test-slave-s3n3 krb5kdc[6684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:07:25.056046 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:25.041650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.6.2.195:45889)
0924 10:07:25.041810 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:25.041814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:25.041836 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:07:25.041993 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:25.041997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:25.042070 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:25.042158 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:25.043425 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:25.044196 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:25.045485 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:25.045891 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:25.045989 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:25.050124 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:25.050139 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:25.050167 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:25.050208 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:25.053032 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:25.053728 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:25.053736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:25.053739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:25.053879 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:25.054320 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:25.054324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:25.054326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:25.054717 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:25.054946 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:25.055585 (+ 639us) server_negotiation.cc:299] Negotiation successful
0924 10:07:25.055771 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:07:25.056713 7180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:25.145251 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 7057
I20260924 10:07:25.161489 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 6697
I20260924 10:07:25.162748 6759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:25.271245 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 6697
2026-09-24T10:07:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4147 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:07:25 dist-test-slave-s3n3 krb5kdc[7233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:07:25 dist-test-slave-s3n3 krb5kdc[7233](info): set up 2 sockets
Sep 24 10:07:25 dist-test-slave-s3n3 krb5kdc[7233](info): commencing operation
krb5kdc: starting...
W20260924 10:07:26.351328 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:07:26 dist-test-slave-s3n3 krb5kdc[7233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:07:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:07:26.535468 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36433
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:36433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:26.855237 7246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:26.855526 7246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:26.855684 7246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:26.862882 7246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:26.862991 7246 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:07:26.863032 7246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:26.863222 7246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:26.863293 7246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:26.863400 7246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:26.874864 7246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:36433
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:26.876238 7246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:26.877647 7246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:26.889300 7252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:26.889477 7251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:26.891919 7254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:26.892323 7246 server_base.cc:1034] running on GCE node
I20260924 10:07:26.893194 7246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:26.894716 7246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:26.895984 7246 hybrid_clock.cc:648] HybridClock initialized: now 1790244446895904 us; error 60 us; skew 500 ppm
I20260924 10:07:26.896360 7246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:26 dist-test-slave-s3n3 krb5kdc[7233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244446, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:26.899775 7246 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:07:26.901324 7246 webserver.cc:460] Webserver started at http://127.6.2.254:34749/ using document root <none> and password file <none>
I20260924 10:07:26.902079 7246 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:26.902165 7246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:26.902501 7246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:26.905639 7246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "00d10669f9904852a09828c543c448be"
format_stamp: "Formatted at 2026-09-24 10:07:26 on dist-test-slave-s3n3"
server_key: "9b86cb2b7b3fbf513ca914620aa4a74a"
server_key_iv: "f4676664301efd6dadeda6912c6ed94d"
server_key_version: "encryptionkey@0"
I20260924 10:07:26.906473 7246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "00d10669f9904852a09828c543c448be"
format_stamp: "Formatted at 2026-09-24 10:07:26 on dist-test-slave-s3n3"
server_key: "9b86cb2b7b3fbf513ca914620aa4a74a"
server_key_iv: "f4676664301efd6dadeda6912c6ed94d"
server_key_version: "encryptionkey@0"
I20260924 10:07:26.911715 7246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260924 10:07:26.915187 7261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:26.916510 7246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:26.916676 7246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "00d10669f9904852a09828c543c448be"
format_stamp: "Formatted at 2026-09-24 10:07:26 on dist-test-slave-s3n3"
server_key: "9b86cb2b7b3fbf513ca914620aa4a74a"
server_key_iv: "f4676664301efd6dadeda6912c6ed94d"
server_key_version: "encryptionkey@0"
I20260924 10:07:26.916903 7246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:26.935364 7246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:26.939760 7246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:26.940069 7246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:26.962806 7246 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:36433
I20260924 10:07:26.962828 7322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:36433 every 8 connection(s)
I20260924 10:07:26.964295 7246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:26.968353 7323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:26.972980 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 7246
I20260924 10:07:26.973230 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:07:26.973946 6155 external_mini_cluster.cc:1428] Setting key b1ace1015115957b16833e48208e8d60
I20260924 10:07:26.980795 7323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be: Bootstrap starting.
I20260924 10:07:26.984803 7323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:26.985970 7323 log.cc:824] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:26.989202 7323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be: No bootstrap required, opened a new log
Sep 24 10:07:26 dist-test-slave-s3n3 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:27.000442 7323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d10669f9904852a09828c543c448be" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36433 } }
I20260924 10:07:27.001065 7323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:27.001180 7323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d10669f9904852a09828c543c448be, State: Initialized, Role: FOLLOWER
I20260924 10:07:27.002010 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:26.976154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.6.2.254:36433)
0924 10:07:26.976999 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:26.977020 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:26.977091 (+ 71us) server_negotiation.cc:407] Connection header received
0924 10:07:26.979577 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:26.979606 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:26.980099 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:26.980540 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:26.982004 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:26.984278 (+ 2274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:26.985682 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:26.986806 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:26.986961 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:26.991797 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:26.991812 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:26.991839 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:26.991895 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:26.996119 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:26.996969 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:26.996978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:26.996988 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:26.997349 (+ 361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:26.997809 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:26.997819 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:26.997823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:26.998704 (+ 881us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:26.999027 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:26.999728 (+ 701us) server_negotiation.cc:299] Negotiation successful
0924 10:07:27.000191 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":368,"threads_started":1}
I20260924 10:07:27.002343 7323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d10669f9904852a09828c543c448be" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36433 } }
I20260924 10:07:27.002763 7323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:27.002939 7323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:27.003185 7323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:27.005715 7323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d10669f9904852a09828c543c448be" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36433 } }
I20260924 10:07:27.006389 7323 leader_election.cc:302] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00d10669f9904852a09828c543c448be; no voters:
I20260924 10:07:27.006928 7323 leader_election.cc:288] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:27.007107 7328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:27.008284 7328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [term 1 LEADER]: Becoming Leader. State: Replica: 00d10669f9904852a09828c543c448be, State: Running, Role: LEADER
I20260924 10:07:27.008976 7328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d10669f9904852a09828c543c448be" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36433 } }
I20260924 10:07:27.009244 7323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:27.012516 7330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00d10669f9904852a09828c543c448be. Latest consensus state: current_term: 1 leader_uuid: "00d10669f9904852a09828c543c448be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d10669f9904852a09828c543c448be" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36433 } } }
I20260924 10:07:27.013307 7330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:27.012914 7329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00d10669f9904852a09828c543c448be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d10669f9904852a09828c543c448be" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36433 } } }
I20260924 10:07:27.014248 7329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:27.015007 7334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:27.034870 7334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:27.047622 7334 catalog_manager.cc:1347] Generated new cluster ID: fed272f5d4654b988498e0ccd4d95b06
I20260924 10:07:27.047741 7334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:27.064814 7334 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:27.067097 7334 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:27.077682 7334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be: Generated new TSK 0
I20260924 10:07:27.078790 7334 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:07:27.122994 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36433
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:27.452095 7349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:27.452437 7349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:27.452607 7349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:27.459947 7349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:27.460114 7349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:27.460278 7349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:07:27.460418 7349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:07:27.470516 7349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.2.254:36433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:27.471922 7349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:27.473264 7349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:27.486864 7354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:27.486970 7357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:27.487087 7355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:27.487911 7349 server_base.cc:1034] running on GCE node
I20260924 10:07:27.488523 7349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:27.489482 7349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:27.490917 7349 hybrid_clock.cc:648] HybridClock initialized: now 1790244447490833 us; error 59 us; skew 500 ppm
I20260924 10:07:27.491370 7349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:27 dist-test-slave-s3n3 krb5kdc[7233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244447, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:27.494578 7349 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:07:27.496142 7349 webserver.cc:460] Webserver started at http://127.6.2.193:42069/ using document root <none> and password file <none>
I20260924 10:07:27.496856 7349 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:27.496987 7349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:27.497336 7349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:27.500355 7349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d"
format_stamp: "Formatted at 2026-09-24 10:07:27 on dist-test-slave-s3n3"
server_key: "ff7b7ef229c9f91191a9036579a7db55"
server_key_iv: "7174b94128d600ecf70c6e977ad8e86b"
server_key_version: "encryptionkey@0"
I20260924 10:07:27.501139 7349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d"
format_stamp: "Formatted at 2026-09-24 10:07:27 on dist-test-slave-s3n3"
server_key: "ff7b7ef229c9f91191a9036579a7db55"
server_key_iv: "7174b94128d600ecf70c6e977ad8e86b"
server_key_version: "encryptionkey@0"
I20260924 10:07:27.506304 7349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:07:27.509955 7364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:27.511593 7349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 10:07:27.511776 7349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d"
format_stamp: "Formatted at 2026-09-24 10:07:27 on dist-test-slave-s3n3"
server_key: "ff7b7ef229c9f91191a9036579a7db55"
server_key_iv: "7174b94128d600ecf70c6e977ad8e86b"
server_key_version: "encryptionkey@0"
I20260924 10:07:27.512018 7349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:27.528390 7349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:27.532562 7349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:27.532855 7349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:27.535701 7349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:27.535811 7349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:27.536011 7349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:27.536091 7349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:07:27 dist-test-slave-s3n3 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244447, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:27.561473 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:27.536302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:39795 (local address 127.6.2.254:36433)
0924 10:07:27.536975 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:27.536981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:27.538163 (+ 1182us) server_negotiation.cc:407] Connection header received
0924 10:07:27.541443 (+ 3280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:27.541452 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:27.541545 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:27.541678 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:27.544226 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:27.545369 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:27.547863 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:27.548486 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:27.548620 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:27.553338 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:27.553346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:27.553352 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:27.553398 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:27.557357 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:27.558476 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:27.558482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:27.558486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:27.558663 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:27.559372 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:27.559379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:27.559384 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:27.559728 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:27.559895 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:27.561038 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0924 10:07:27.561214 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":427,"threads_started":1}
I20260924 10:07:27.563637 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:27.536958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36433 (local address 127.6.2.193:39795)
0924 10:07:27.537960 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:27.538022 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:27.541066 (+ 3044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:27.542072 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:27.542090 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:27.542962 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:27.544009 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:27.544044 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:27.545556 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:27.545564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:27.547676 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:27.547688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:27.548757 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:27.548765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:27.548986 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:27.549726 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:27.549748 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:27.553004 (+ 3256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:27.557551 (+ 4547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:27.557562 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:27.557586 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:27.558333 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:27.558811 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:27.558820 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:27.558826 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:27.559209 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:27.559873 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:27.559882 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:27.560525 (+ 643us) client_negotiation.cc:769] Sending connection context
0924 10:07:27.560949 (+ 424us) client_negotiation.cc:240] Negotiation successful
0924 10:07:27.561667 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":396,"threads_started":1}
I20260924 10:07:27.589740 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:27.574644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:49027 (local address 127.6.2.254:36433)
0924 10:07:27.574818 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:27.574823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:27.575170 (+ 347us) server_negotiation.cc:407] Connection header received
0924 10:07:27.575482 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:27.575490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:27.575587 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:27.575700 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:27.577394 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:27.578387 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:27.580895 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:27.581495 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:27.581611 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:27.584323 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:27.584328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:27.584331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:27.584361 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:27.587034 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:27.587774 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:27.587781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:27.587787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:27.587928 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:27.588312 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:27.588317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:27.588320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:27.588587 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:27.588739 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:27.589128 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 10:07:27.589361 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:07:27.589845 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:27.574177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36433 (local address 127.6.2.193:49027)
0924 10:07:27.575010 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:27.575032 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:27.575287 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:27.575766 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:27.575774 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:27.576574 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:27.577045 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:27.577074 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:27.578558 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:27.578563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:27.580607 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:27.580629 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:27.581714 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:27.581724 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:27.581980 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:27.582719 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:27.582751 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:27.584099 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:27.587227 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:27.587232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:27.587236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:27.587636 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:27.588044 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:27.588049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:27.588052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:27.588189 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:27.588707 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:27.588712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:27.588892 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 10:07:27.589029 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 10:07:27.589406 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":545,"threads_started":1}
I20260924 10:07:27.596566 7349 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:35093
I20260924 10:07:27.596606 7484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:35093 every 8 connection(s)
I20260924 10:07:27.597776 7349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:07:27.599779 7485 heartbeater.cc:347] Connected to a master server at 127.6.2.254:36433
I20260924 10:07:27.599975 7485 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:27.600554 7485 heartbeater.cc:511] Master 127.6.2.254:36433 requested a full tablet report, sending...
I20260924 10:07:27.602430 7278 ts_manager.cc:194] Registered new tserver with Master: 6e3fea1ff0f54e9b8220ae8f36c0006d (127.6.2.193:35093)
I20260924 10:07:27.602958 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 7349
I20260924 10:07:27.603176 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:07:27.603825 6155 external_mini_cluster.cc:1428] Setting key d55154d803e3d33bbb83294f538df17f
I20260924 10:07:27.604059 7278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:39795
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:07:27.647132 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36433
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:27.969879 7489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:27.970139 7489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:27.970304 7489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:27.977902 7489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:27.978011 7489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:27.978149 7489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:07:27.978281 7489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:07:27.988082 7489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.2.254:36433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:27.989347 7489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:27.990670 7489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:28.003767 7495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:28.004442 7489 server_base.cc:1034] running on GCE node
W20260924 10:07:28.004518 7494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:28.004693 7497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:28.005338 7489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:28.006312 7489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:28.007544 7489 hybrid_clock.cc:648] HybridClock initialized: now 1790244448007469 us; error 53 us; skew 500 ppm
I20260924 10:07:28.007840 7489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:28 dist-test-slave-s3n3 krb5kdc[7233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244448, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:28.015727 7489 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:07:28.017261 7489 webserver.cc:460] Webserver started at http://127.6.2.194:45429/ using document root <none> and password file <none>
I20260924 10:07:28.017968 7489 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:28.018100 7489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:28.018437 7489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:28.021524 7489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "9a476336c7544b45af7c4afa944b6388"
format_stamp: "Formatted at 2026-09-24 10:07:28 on dist-test-slave-s3n3"
server_key: "92d391123bc839c297fe4a1d14191b7f"
server_key_iv: "521ce67f199aa7aaf814afe2c11c3019"
server_key_version: "encryptionkey@0"
I20260924 10:07:28.022310 7489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "9a476336c7544b45af7c4afa944b6388"
format_stamp: "Formatted at 2026-09-24 10:07:28 on dist-test-slave-s3n3"
server_key: "92d391123bc839c297fe4a1d14191b7f"
server_key_iv: "521ce67f199aa7aaf814afe2c11c3019"
server_key_version: "encryptionkey@0"
I20260924 10:07:28.027552 7489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 10:07:28.031546 7504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:28.032905 7489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:07:28.033087 7489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "9a476336c7544b45af7c4afa944b6388"
format_stamp: "Formatted at 2026-09-24 10:07:28 on dist-test-slave-s3n3"
server_key: "92d391123bc839c297fe4a1d14191b7f"
server_key_iv: "521ce67f199aa7aaf814afe2c11c3019"
server_key_version: "encryptionkey@0"
I20260924 10:07:28.033310 7489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:28.050261 7489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:28.054739 7489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:28.055050 7489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:28.058166 7489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:28.058313 7489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:28.058463 7489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:28.058573 7489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:07:28 dist-test-slave-s3n3 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244448, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:28.084622 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.058826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:37827 (local address 127.6.2.254:36433)
0924 10:07:28.059020 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:28.059026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:28.060472 (+ 1446us) server_negotiation.cc:407] Connection header received
0924 10:07:28.062514 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:28.062523 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:28.062668 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:28.062807 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:28.065130 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.066478 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.070518 (+ 4040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.071242 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.071394 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.076820 (+ 5426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:28.076828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:28.076837 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:28.076884 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:28.080373 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:28.081512 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:28.081520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:28.081526 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:28.081720 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:28.082391 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:28.082398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:28.082404 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:28.082815 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:28.082989 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:28.084054 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0924 10:07:28.084289 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:07:28.086227 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.059354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36433 (local address 127.6.2.194:37827)
0924 10:07:28.060267 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:28.060314 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:28.062226 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:28.063125 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:28.063137 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:28.063929 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:28.064908 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.064937 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.066721 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.066730 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.069962 (+ 3232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.069981 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.071567 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.071577 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.071995 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.072833 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:28.072872 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:28.076528 (+ 3656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:28.080530 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:28.080542 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:28.080561 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:28.081330 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:28.081868 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:28.081876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:28.081883 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:28.082221 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:28.082992 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:28.083002 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:28.083481 (+ 479us) client_negotiation.cc:769] Sending connection context
0924 10:07:28.083884 (+ 403us) client_negotiation.cc:240] Negotiation successful
0924 10:07:28.084397 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":401,"threads_started":1}
I20260924 10:07:28.108672 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.091780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:36253 (local address 127.6.2.254:36433)
0924 10:07:28.091979 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:28.091986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:28.092769 (+ 783us) server_negotiation.cc:407] Connection header received
0924 10:07:28.093007 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:28.093012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:28.093071 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:28.093167 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:28.094593 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.095573 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.097769 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.098140 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.098245 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.101476 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:28.101483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:28.101487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:28.101522 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:28.105580 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:28.106260 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:28.106268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:28.106274 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:28.106457 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:28.106815 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:28.106823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:28.106828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:28.107361 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:28.107524 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:28.108210 (+ 686us) server_negotiation.cc:299] Negotiation successful
0924 10:07:28.108447 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:07:28.109195 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.091945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36433 (local address 127.6.2.194:36253)
0924 10:07:28.092630 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:28.092661 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:28.092839 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:28.093219 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:28.093227 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:28.093963 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:28.094418 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.094434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.095738 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.095745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.097596 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.097616 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.098355 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.098364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.098622 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.099668 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:28.099706 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:28.101228 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:28.105750 (+ 4522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:28.105754 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:28.105757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:28.106141 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:28.106583 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:28.106586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:28.106589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:28.106702 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:28.107598 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:28.107607 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:28.107876 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 10:07:28.108094 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 10:07:28.108523 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":329,"threads_started":1}
I20260924 10:07:28.116598 7489 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:43875
I20260924 10:07:28.116639 7623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:43875 every 8 connection(s)
I20260924 10:07:28.117779 7489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:07:28.119702 7624 heartbeater.cc:347] Connected to a master server at 127.6.2.254:36433
I20260924 10:07:28.119899 7624 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:28.120524 7624 heartbeater.cc:511] Master 127.6.2.254:36433 requested a full tablet report, sending...
I20260924 10:07:28.121865 7278 ts_manager.cc:194] Registered new tserver with Master: 9a476336c7544b45af7c4afa944b6388 (127.6.2.194:43875)
I20260924 10:07:28.122715 7278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:37827
I20260924 10:07:28.125586 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 7489
I20260924 10:07:28.125778 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:07:28.126258 6155 external_mini_cluster.cc:1428] Setting key b8f9bb3811e213e8bdd460373e333155
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:07:28.167569 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36433
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:28.486667 7628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:28.487004 7628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:28.487191 7628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:28.494531 7628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:28.494699 7628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:28.494844 7628 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:07:28.494982 7628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:07:28.504870 7628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.2.254:36433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:28.506183 7628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:28.507577 7628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:28.520678 7634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:28.521082 7636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:28.521195 7633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:28.521924 7628 server_base.cc:1034] running on GCE node
I20260924 10:07:28.522591 7628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:28.523555 7628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:28.524825 7628 hybrid_clock.cc:648] HybridClock initialized: now 1790244448524762 us; error 37 us; skew 500 ppm
I20260924 10:07:28.525159 7628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:28 dist-test-slave-s3n3 krb5kdc[7233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244448, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:28.528394 7628 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:07:28.529755 7628 webserver.cc:460] Webserver started at http://127.6.2.195:34451/ using document root <none> and password file <none>
I20260924 10:07:28.530402 7628 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:28.530483 7628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:28.530748 7628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:28.533646 7628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "9d02767a65b34c91a4bb29f5dfaa03de"
format_stamp: "Formatted at 2026-09-24 10:07:28 on dist-test-slave-s3n3"
server_key: "cebc0aa073965586f1f64f8bb473c683"
server_key_iv: "ae75899876d69cc91875a317dedcc608"
server_key_version: "encryptionkey@0"
I20260924 10:07:28.534438 7628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "9d02767a65b34c91a4bb29f5dfaa03de"
format_stamp: "Formatted at 2026-09-24 10:07:28 on dist-test-slave-s3n3"
server_key: "cebc0aa073965586f1f64f8bb473c683"
server_key_iv: "ae75899876d69cc91875a317dedcc608"
server_key_version: "encryptionkey@0"
I20260924 10:07:28.539845 7628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:07:28.543457 7643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:28.544763 7628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:07:28.544950 7628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "9d02767a65b34c91a4bb29f5dfaa03de"
format_stamp: "Formatted at 2026-09-24 10:07:28 on dist-test-slave-s3n3"
server_key: "cebc0aa073965586f1f64f8bb473c683"
server_key_iv: "ae75899876d69cc91875a317dedcc608"
server_key_version: "encryptionkey@0"
I20260924 10:07:28.545236 7628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:28.560529 7628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:28.564715 7628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:28.564931 7628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:28.567631 7628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:28.567736 7628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:28.567879 7628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:28.567945 7628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:07:28 dist-test-slave-s3n3 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244448, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:28.591856 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.568294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:38011 (local address 127.6.2.254:36433)
0924 10:07:28.568509 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:28.568515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:28.570231 (+ 1716us) server_negotiation.cc:407] Connection header received
0924 10:07:28.572294 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:28.572300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:28.572389 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:28.572496 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:28.574774 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.575786 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.578528 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.578938 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.579070 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.584508 (+ 5438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:28.584517 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:28.584524 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:28.584568 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:28.588486 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:28.589464 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:28.589472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:28.589477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:28.589619 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:28.590065 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:28.590070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:28.590073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:28.590365 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:28.590537 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:28.591270 (+ 733us) server_negotiation.cc:299] Negotiation successful
0924 10:07:28.591500 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 10:07:28.593019 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.568913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36433 (local address 127.6.2.195:38011)
0924 10:07:28.570051 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:28.570090 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:28.572024 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:28.572816 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:28.572828 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:28.573661 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:28.574587 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.574609 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.576005 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.576009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.578337 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.578356 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.579230 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.579243 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.579608 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.580478 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:28.580531 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:28.584194 (+ 3663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:28.588698 (+ 4504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:28.588712 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:28.588733 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:28.589342 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:28.589730 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:28.589737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:28.589740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:28.589946 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:28.590491 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:28.590501 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:28.590877 (+ 376us) client_negotiation.cc:769] Sending connection context
0924 10:07:28.591192 (+ 315us) client_negotiation.cc:240] Negotiation successful
0924 10:07:28.591741 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":663,"threads_started":1}
I20260924 10:07:28.606978 7485 heartbeater.cc:503] Master 127.6.2.254:36433 was elected leader, sending a full tablet report...
I20260924 10:07:28.623494 7628 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:41071
I20260924 10:07:28.623528 7762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:41071 every 8 connection(s)
I20260924 10:07:28.624840 7628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:07:28.626973 7763 heartbeater.cc:347] Connected to a master server at 127.6.2.254:36433
I20260924 10:07:28.627225 7763 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:28.628077 7763 heartbeater.cc:511] Master 127.6.2.254:36433 requested a full tablet report, sending...
I20260924 10:07:28.628095 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 7628
I20260924 10:07:28.628338 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:07:28.628896 6155 external_mini_cluster.cc:1428] Setting key e496208a59bc7facdbdc65a19e59eca9
I20260924 10:07:28.629499 7278 ts_manager.cc:194] Registered new tserver with Master: 9d02767a65b34c91a4bb29f5dfaa03de (127.6.2.195:41071)
I20260924 10:07:28.629602 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.614029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:37429 (local address 127.6.2.254:36433)
0924 10:07:28.614269 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:28.614278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:28.614361 (+ 83us) server_negotiation.cc:407] Connection header received
0924 10:07:28.614621 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:28.614629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:28.614717 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:28.614824 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:28.616197 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.617248 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.619860 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.620948 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.621094 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.623620 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:28.623629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:28.623636 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:28.623677 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:28.626650 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:28.627423 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:28.627428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:28.627432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:28.627584 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:28.628061 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:28.628068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:28.628074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:28.628411 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:28.628558 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:28.629006 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 10:07:28.629226 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 10:07:28.629647 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.613540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36433 (local address 127.6.2.195:37429)
0924 10:07:28.614231 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:28.614255 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:28.614475 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:28.614854 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:28.614859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:28.615586 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:28.616004 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.616034 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.617418 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.617427 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.619682 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.619700 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.621179 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.621189 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.621420 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.622060 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:28.622083 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:28.623419 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:28.626774 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:28.626779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:28.626783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:28.627270 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:28.627713 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:28.627718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:28.627720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:28.627909 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:28.628525 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:28.628530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:28.628737 (+ 207us) client_negotiation.cc:769] Sending connection context
0924 10:07:28.628883 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 10:07:28.629203 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":546,"threads_started":1}
I20260924 10:07:28.630431 7278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:38011
I20260924 10:07:28.632352 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:07:28 dist-test-slave-s3n3 krb5kdc[7233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:07:28 dist-test-slave-s3n3 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:28.679131 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.664562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.6.2.254:36433)
0924 10:07:28.664782 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:28.664789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:28.664913 (+ 124us) server_negotiation.cc:407] Connection header received
0924 10:07:28.665180 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:28.665185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:28.665260 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:28.665375 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:28.666843 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.667687 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.668924 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.669340 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.669474 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.673258 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:28.673263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:28.673266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:28.673298 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:28.676445 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:28.677129 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:28.677136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:28.677144 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:28.677293 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:28.677675 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:28.677681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:28.677684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:28.677943 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:28.678183 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:28.678488 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 10:07:28.678708 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 10:07:28.683461 7277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:07:28.690472 7277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:07:28.716599 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.702539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:41071 (local address 127.0.0.1:43176)
0924 10:07:28.704848 (+ 2309us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:28.704883 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:28.705050 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:28.706099 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:28.706110 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:28.706152 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:28.706712 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.706733 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.708981 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.708986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.711960 (+ 2974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.711973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.715517 (+ 3544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.715527 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.715780 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.715862 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 10:07:28.715997 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 10:07:28.716216 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2062,"thread_start_us":13209,"threads_started":1}
I20260924 10:07:28.716831 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.702924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.6.2.195:41071)
0924 10:07:28.704416 (+ 1492us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:28.704428 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:28.705236 (+ 808us) server_negotiation.cc:407] Connection header received
0924 10:07:28.705349 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:28.705357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:28.705609 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:28.705786 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:28.706997 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.708829 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.712161 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.714102 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.715997 (+ 1895us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.716262 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:28.716337 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 10:07:28.716509 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1249,"thread_start_us":1025,"threads_started":1}
I20260924 10:07:28.721220 7693 tablet_service.cc:1467] Processing CreateTablet for tablet 4924c3eda3d749ff9cad80bc2b1fb055 (DEFAULT_TABLE table=test-table [id=28a9a18d207f4bf9bd0612fa847297d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:28.722906 7693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4924c3eda3d749ff9cad80bc2b1fb055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:28.723973 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.703011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.6.2.194:43875)
0924 10:07:28.715548 (+ 12537us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:28.715560 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:28.716396 (+ 836us) server_negotiation.cc:407] Connection header received
0924 10:07:28.716671 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:28.716681 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:28.717244 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:28.717460 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:28.718266 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.719256 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.721777 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.722691 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.722850 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.723047 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:28.723526 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 10:07:28.723675 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12367,"thread_start_us":12174,"threads_started":1}
I20260924 10:07:28.724015 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.702724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:43875 (local address 127.0.0.1:56858)
0924 10:07:28.716181 (+ 13457us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:28.716201 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:28.716469 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:28.717670 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:28.717675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:28.717698 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:28.718074 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.718085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.719441 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.719444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.721570 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.721586 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.722920 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.722927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.723330 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.723357 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:07:28.723474 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 10:07:28.723607 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13187,"mutex_wait_us":40,"thread_start_us":1877,"threads_started":1}
I20260924 10:07:28.726261 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.704591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:35093 (local address 127.0.0.1:48332)
0924 10:07:28.716856 (+ 12265us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:28.716878 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:28.717106 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:28.719617 (+ 2511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:28.719621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:28.719645 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:28.719869 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.719879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.721818 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.721823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.723698 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.723712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.725529 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.725536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.725712 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.725762 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 10:07:28.725853 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 10:07:28.725987 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12135,"mutex_wait_us":737,"thread_start_us":13936,"threads_started":1}
I20260924 10:07:28.726905 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.703587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.6.2.193:35093)
0924 10:07:28.714796 (+ 11209us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:28.714809 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:28.716969 (+ 2160us) server_negotiation.cc:407] Connection header received
0924 10:07:28.717325 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:28.717336 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:28.717612 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:28.717797 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:28.720038 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.721664 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.723910 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.725355 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.725499 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.725686 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:28.725990 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 10:07:28.726171 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10997,"thread_start_us":11262,"threads_started":1}
I20260924 10:07:28.728183 7554 tablet_service.cc:1467] Processing CreateTablet for tablet 4924c3eda3d749ff9cad80bc2b1fb055 (DEFAULT_TABLE table=test-table [id=28a9a18d207f4bf9bd0612fa847297d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:28.729292 7554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4924c3eda3d749ff9cad80bc2b1fb055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:28.730363 7415 tablet_service.cc:1467] Processing CreateTablet for tablet 4924c3eda3d749ff9cad80bc2b1fb055 (DEFAULT_TABLE table=test-table [id=28a9a18d207f4bf9bd0612fa847297d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:28.731684 7415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4924c3eda3d749ff9cad80bc2b1fb055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:28.737897 7782 tablet_bootstrap.cc:492] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de: Bootstrap starting.
I20260924 10:07:28.741298 7783 tablet_bootstrap.cc:492] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388: Bootstrap starting.
I20260924 10:07:28.742081 7782 tablet_bootstrap.cc:654] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:28.743922 7782 log.cc:824] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:28.744585 7783 tablet_bootstrap.cc:654] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:28.745878 7783 log.cc:824] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:28.746852 7782 tablet_bootstrap.cc:492] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de: No bootstrap required, opened a new log
I20260924 10:07:28.746850 7785 tablet_bootstrap.cc:492] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Bootstrap starting.
I20260924 10:07:28.747167 7782 ts_tablet_manager.cc:1397] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260924 10:07:28.748853 7783 tablet_bootstrap.cc:492] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388: No bootstrap required, opened a new log
I20260924 10:07:28.749126 7783 ts_tablet_manager.cc:1397] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 10:07:28.749922 7785 tablet_bootstrap.cc:654] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:28.750855 7785 log.cc:824] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:28.753223 7785 tablet_bootstrap.cc:492] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d: No bootstrap required, opened a new log
I20260924 10:07:28.753470 7785 ts_tablet_manager.cc:1397] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 10:07:28.757828 7782 raft_consensus.cc:348] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } }
I20260924 10:07:28.758279 7782 raft_consensus.cc:374] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:28.758374 7782 raft_consensus.cc:729] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d02767a65b34c91a4bb29f5dfaa03de, State: Initialized, Role: FOLLOWER
I20260924 10:07:28.758978 7782 consensus_queue.cc:260] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } }
I20260924 10:07:28.759407 7783 raft_consensus.cc:348] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } }
I20260924 10:07:28.759846 7783 raft_consensus.cc:374] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:28.759946 7783 raft_consensus.cc:729] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a476336c7544b45af7c4afa944b6388, State: Initialized, Role: FOLLOWER
I20260924 10:07:28.760313 7763 heartbeater.cc:503] Master 127.6.2.254:36433 was elected leader, sending a full tablet report...
I20260924 10:07:28.760484 7783 consensus_queue.cc:260] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } }
I20260924 10:07:28.761497 7624 heartbeater.cc:503] Master 127.6.2.254:36433 was elected leader, sending a full tablet report...
I20260924 10:07:28.761734 7785 raft_consensus.cc:348] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } }
I20260924 10:07:28.762104 7785 raft_consensus.cc:374] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:28.762210 7785 raft_consensus.cc:729] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3fea1ff0f54e9b8220ae8f36c0006d, State: Initialized, Role: FOLLOWER
I20260924 10:07:28.762256 7783 ts_tablet_manager.cc:1428] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388: Time spent starting tablet: real 0.013s user 0.005s sys 0.010s
I20260924 10:07:28.762774 7785 consensus_queue.cc:260] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } }
I20260924 10:07:28.763851 7782 ts_tablet_manager.cc:1428] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
I20260924 10:07:28.766122 7785 ts_tablet_manager.cc:1428] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260924 10:07:28.850176 7486 tablet.cc:2369] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:07:28.869807 7625 tablet.cc:2369] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:07:28.876646 7764 tablet.cc:2369] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:07:28.907318 7789 raft_consensus.cc:482] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:07:28.907688 7789 raft_consensus.cc:504] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } }
I20260924 10:07:28.909556 7789 leader_election.cc:288] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d02767a65b34c91a4bb29f5dfaa03de (127.6.2.195:41071), 6e3fea1ff0f54e9b8220ae8f36c0006d (127.6.2.193:35093)
I20260924 10:07:28.917447 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.910082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:58291 (local address 127.6.2.193:35093)
0924 10:07:28.910518 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:28.910527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:28.911285 (+ 758us) server_negotiation.cc:407] Connection header received
0924 10:07:28.911537 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:28.911546 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:28.911698 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:28.911824 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:28.912282 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.913240 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.915580 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.916454 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.916631 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.916740 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:28.917050 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 10:07:28.917198 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260924 10:07:28.917564 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.910055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:54097 (local address 127.6.2.195:41071)
0924 10:07:28.910309 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:28.910318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:28.910870 (+ 552us) server_negotiation.cc:407] Connection header received
0924 10:07:28.911137 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:28.911141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:28.911203 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:28.911314 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:28.911825 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.912876 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.915561 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.916593 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.916752 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.916863 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:28.917185 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 10:07:28.917314 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 10:07:28.917630 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.909987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:35093 (local address 127.6.2.194:58291)
0924 10:07:28.911176 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:28.911191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:28.911331 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:28.911863 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:28.911865 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:28.911883 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:28.912142 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.912150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.913369 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.913374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.915449 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.915460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.916694 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.916699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.916885 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.916909 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:07:28.917015 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 10:07:28.917158 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"mutex_wait_us":23,"thread_start_us":416,"threads_started":1}
I20260924 10:07:28.917712 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.909861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:41071 (local address 127.6.2.194:54097)
0924 10:07:28.910748 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:28.910767 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:28.910953 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:28.911346 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:28.911351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:28.911388 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:28.911693 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.911701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.912999 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.913003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.915290 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:28.915325 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.916825 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.916832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:28.917015 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.917038 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:07:28.917161 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 10:07:28.917291 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":2,"thread_start_us":952,"threads_started":1}
I20260924 10:07:28.918763 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4924c3eda3d749ff9cad80bc2b1fb055" candidate_uuid: "9a476336c7544b45af7c4afa944b6388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" is_pre_election: true
I20260924 10:07:28.918891 7439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4924c3eda3d749ff9cad80bc2b1fb055" candidate_uuid: "9a476336c7544b45af7c4afa944b6388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" is_pre_election: true
I20260924 10:07:28.919374 7713 raft_consensus.cc:2454] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a476336c7544b45af7c4afa944b6388 in term 0.
I20260924 10:07:28.919371 7439 raft_consensus.cc:2454] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a476336c7544b45af7c4afa944b6388 in term 0.
I20260924 10:07:28.920027 7506 leader_election.cc:302] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6e3fea1ff0f54e9b8220ae8f36c0006d, 9a476336c7544b45af7c4afa944b6388, 9d02767a65b34c91a4bb29f5dfaa03de; no voters:
I20260924 10:07:28.920450 7789 raft_consensus.cc:2781] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:07:28.920594 7789 raft_consensus.cc:482] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:07:28.920702 7789 raft_consensus.cc:3037] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:28.922946 7789 raft_consensus.cc:504] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } }
I20260924 10:07:28.923642 7789 leader_election.cc:288] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [CANDIDATE]: Term 1 election: Requested vote from peers 9d02767a65b34c91a4bb29f5dfaa03de (127.6.2.195:41071), 6e3fea1ff0f54e9b8220ae8f36c0006d (127.6.2.193:35093)
I20260924 10:07:28.924060 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4924c3eda3d749ff9cad80bc2b1fb055" candidate_uuid: "9a476336c7544b45af7c4afa944b6388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d02767a65b34c91a4bb29f5dfaa03de"
I20260924 10:07:28.924099 7439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4924c3eda3d749ff9cad80bc2b1fb055" candidate_uuid: "9a476336c7544b45af7c4afa944b6388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d"
I20260924 10:07:28.924263 7713 raft_consensus.cc:3037] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:28.924295 7439 raft_consensus.cc:3037] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:28.926384 7439 raft_consensus.cc:2454] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a476336c7544b45af7c4afa944b6388 in term 1.
I20260924 10:07:28.926412 7713 raft_consensus.cc:2454] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a476336c7544b45af7c4afa944b6388 in term 1.
I20260924 10:07:28.926798 7506 leader_election.cc:302] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fea1ff0f54e9b8220ae8f36c0006d, 9a476336c7544b45af7c4afa944b6388; no voters:
I20260924 10:07:28.927137 7789 raft_consensus.cc:2781] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:28.927590 7789 raft_consensus.cc:686] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [term 1 LEADER]: Becoming Leader. State: Replica: 9a476336c7544b45af7c4afa944b6388, State: Running, Role: LEADER
I20260924 10:07:28.928150 7789 consensus_queue.cc:237] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } }
I20260924 10:07:28.933018 7276 catalog_manager.cc:5495] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a476336c7544b45af7c4afa944b6388 (127.6.2.194). New cstate: current_term: 1 leader_uuid: "9a476336c7544b45af7c4afa944b6388" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } health_report { overall_health: UNKNOWN } } }
I20260924 10:07:28.959178 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:28.953584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.6.2.194:43875)
0924 10:07:28.953812 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:28.953819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:28.953951 (+ 132us) server_negotiation.cc:407] Connection header received
0924 10:07:28.954334 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:28.954338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:28.954423 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:28.954568 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:28.954912 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.955691 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.956985 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:28.957398 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:28.957539 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:28.958023 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:28.958518 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:28.958678 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:28.958764 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 10:07:28.958844 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 10:07:28.967998 7439 raft_consensus.cc:1261] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 1 FOLLOWER]: Refusing update from remote peer 9a476336c7544b45af7c4afa944b6388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:28.968191 7713 raft_consensus.cc:1261] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de [term 1 FOLLOWER]: Refusing update from remote peer 9a476336c7544b45af7c4afa944b6388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:28.968966 7789 consensus_queue.cc:1035] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:28.969281 7793 consensus_queue.cc:1035] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:28.984918 7798 mvcc.cc:205] Tried to move back new op lower bound from 7332841262962384896 to 7332841262813507584. Current Snapshot: MvccSnapshot[applied={T|T < 7332841262962384896}]
I20260924 10:07:29.003160 7276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:29.003449 7276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:29.007252 7276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be: Sending DeleteTablet for 3 replicas of tablet 4924c3eda3d749ff9cad80bc2b1fb055
I20260924 10:07:29.008396 7693 tablet_service.cc:1514] Processing DeleteTablet for tablet 4924c3eda3d749ff9cad80bc2b1fb055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:29 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:43176
I20260924 10:07:29.008399 7554 tablet_service.cc:1514] Processing DeleteTablet for tablet 4924c3eda3d749ff9cad80bc2b1fb055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:29 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:56858
I20260924 10:07:29.008627 7415 tablet_service.cc:1514] Processing DeleteTablet for tablet 4924c3eda3d749ff9cad80bc2b1fb055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:29 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:48332
I20260924 10:07:29.011022 7276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:07:29.012291 7276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:07:29.022526 7806 tablet_replica.cc:330] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388: stopping tablet replica
I20260924 10:07:29.022563 7414 tablet_service.cc:1467] Processing CreateTablet for tablet 8d38647a234c4f0a933f462d76f18ad7 (DEFAULT_TABLE table=test-table [id=cc8b1e3431f340c084214b41da6fa276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:29.023222 7414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d38647a234c4f0a933f462d76f18ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:29.023612 7693 tablet_service.cc:1467] Processing CreateTablet for tablet 8d38647a234c4f0a933f462d76f18ad7 (DEFAULT_TABLE table=test-table [id=cc8b1e3431f340c084214b41da6fa276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:29.025694 7693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d38647a234c4f0a933f462d76f18ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:29.030278 7785 tablet_bootstrap.cc:492] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Bootstrap starting.
I20260924 10:07:29.030447 7807 tablet_replica.cc:330] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de: stopping tablet replica
I20260924 10:07:29.032071 7785 tablet_bootstrap.cc:654] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:29.023609 7554 tablet_service.cc:1467] Processing CreateTablet for tablet 8d38647a234c4f0a933f462d76f18ad7 (DEFAULT_TABLE table=test-table [id=cc8b1e3431f340c084214b41da6fa276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:29.035529 7554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d38647a234c4f0a933f462d76f18ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:29.037047 7807 raft_consensus.cc:2229] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:29.037741 7806 raft_consensus.cc:2229] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:07:29.038537 7807 raft_consensus.cc:2258] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:29.039880 7806 raft_consensus.cc:2258] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:29.044363 7783 tablet_bootstrap.cc:492] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388: Bootstrap starting.
I20260924 10:07:29.044751 7808 tablet_replica.cc:330] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d: stopping tablet replica
I20260924 10:07:29.045075 7785 tablet_bootstrap.cc:492] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d: No bootstrap required, opened a new log
I20260924 10:07:29.045243 7785 ts_tablet_manager.cc:1397] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260924 10:07:29.045275 7808 raft_consensus.cc:2229] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:29.045787 7808 raft_consensus.cc:2258] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:29.046196 7783 tablet_bootstrap.cc:654] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:29.046376 7785 raft_consensus.cc:348] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.046690 7785 raft_consensus.cc:374] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:29.046804 7785 raft_consensus.cc:729] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3fea1ff0f54e9b8220ae8f36c0006d, State: Initialized, Role: FOLLOWER
I20260924 10:07:29.047151 7785 consensus_queue.cc:260] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.047731 7808 ts_tablet_manager.cc:1905] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:29.046746 7807 ts_tablet_manager.cc:1905] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:29.049041 7806 ts_tablet_manager.cc:1905] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:29.050091 7785 ts_tablet_manager.cc:1428] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260924 10:07:29.052286 7782 tablet_bootstrap.cc:492] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de: Bootstrap starting.
I20260924 10:07:29.053524 7782 tablet_bootstrap.cc:654] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:29.055495 7783 tablet_bootstrap.cc:492] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388: No bootstrap required, opened a new log
I20260924 10:07:29.055766 7783 ts_tablet_manager.cc:1397] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.008s
I20260924 10:07:29.056908 7783 raft_consensus.cc:348] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.057328 7783 raft_consensus.cc:374] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:29.057499 7783 raft_consensus.cc:729] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a476336c7544b45af7c4afa944b6388, State: Initialized, Role: FOLLOWER
I20260924 10:07:29.057924 7783 consensus_queue.cc:260] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.058606 7783 ts_tablet_manager.cc:1428] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:07:29.060582 7807 ts_tablet_manager.cc:1918] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:07:29.060747 7807 log.cc:1196] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/4924c3eda3d749ff9cad80bc2b1fb055
I20260924 10:07:29.061252 7807 ts_tablet_manager.cc:1939] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9d02767a65b34c91a4bb29f5dfaa03de: Deleting consensus metadata
I20260924 10:07:29.063165 7262 catalog_manager.cc:4841] TS 9d02767a65b34c91a4bb29f5dfaa03de (127.6.2.195:41071): tablet 4924c3eda3d749ff9cad80bc2b1fb055 (table test-table [id=28a9a18d207f4bf9bd0612fa847297d2]) successfully deleted
I20260924 10:07:29.064698 7782 tablet_bootstrap.cc:492] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de: No bootstrap required, opened a new log
I20260924 10:07:29.064729 7806 ts_tablet_manager.cc:1918] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:07:29.064846 7782 ts_tablet_manager.cc:1397] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260924 10:07:29.064895 7806 log.cc:1196] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/4924c3eda3d749ff9cad80bc2b1fb055
I20260924 10:07:29.065431 7806 ts_tablet_manager.cc:1939] T 4924c3eda3d749ff9cad80bc2b1fb055 P 9a476336c7544b45af7c4afa944b6388: Deleting consensus metadata
I20260924 10:07:29.065816 7782 raft_consensus.cc:348] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.066041 7782 raft_consensus.cc:374] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:29.066138 7782 raft_consensus.cc:729] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d02767a65b34c91a4bb29f5dfaa03de, State: Initialized, Role: FOLLOWER
I20260924 10:07:29.066360 7808 ts_tablet_manager.cc:1918] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:07:29.066573 7808 log.cc:1196] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/4924c3eda3d749ff9cad80bc2b1fb055
I20260924 10:07:29.066517 7782 consensus_queue.cc:260] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.067233 7808 ts_tablet_manager.cc:1939] T 4924c3eda3d749ff9cad80bc2b1fb055 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Deleting consensus metadata
I20260924 10:07:29.067241 7264 catalog_manager.cc:4841] TS 9a476336c7544b45af7c4afa944b6388 (127.6.2.194:43875): tablet 4924c3eda3d749ff9cad80bc2b1fb055 (table test-table [id=28a9a18d207f4bf9bd0612fa847297d2]) successfully deleted
I20260924 10:07:29.068666 7782 ts_tablet_manager.cc:1428] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 10:07:29.068941 7263 catalog_manager.cc:4841] TS 6e3fea1ff0f54e9b8220ae8f36c0006d (127.6.2.193:35093): tablet 4924c3eda3d749ff9cad80bc2b1fb055 (table test-table [id=28a9a18d207f4bf9bd0612fa847297d2]) successfully deleted
I20260924 10:07:29.187192 7788 raft_consensus.cc:482] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:07:29.187440 7788 raft_consensus.cc:504] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.189277 7788 leader_election.cc:288] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e3fea1ff0f54e9b8220ae8f36c0006d (127.6.2.193:35093), 9a476336c7544b45af7c4afa944b6388 (127.6.2.194:43875)
I20260924 10:07:29.196920 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.189835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:47033 (local address 127.6.2.193:35093)
0924 10:07:29.190020 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.190026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.191097 (+ 1071us) server_negotiation.cc:407] Connection header received
0924 10:07:29.191277 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.191285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.191377 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.191477 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:29.191976 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.193116 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.194864 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.195810 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.195932 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.196037 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.196493 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.196627 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:07:29.197002 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.189545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:35093 (local address 127.6.2.195:47033)
0924 10:07:29.190925 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:29.190946 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:29.191137 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:29.191525 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:29.191531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:29.191556 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:29.191780 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.191789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.193265 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.193270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.194705 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.194716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.196101 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.196112 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.196326 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.196350 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:07:29.196451 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 10:07:29.196628 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1268,"mutex_wait_us":1,"thread_start_us":1743,"threads_started":1}
I20260924 10:07:29.197654 7439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d38647a234c4f0a933f462d76f18ad7" candidate_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" is_pre_election: true
I20260924 10:07:29.197930 7439 raft_consensus.cc:2454] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d02767a65b34c91a4bb29f5dfaa03de in term 0.
I20260924 10:07:29.198450 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.189545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:43875 (local address 127.6.2.195:34857)
0924 10:07:29.191554 (+ 2009us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:29.191570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:29.191752 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:29.192108 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:29.192113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:29.192135 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:29.192414 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.192422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.193559 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.193565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.195710 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.195725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.197599 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.197605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.197796 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.197833 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 10:07:29.197977 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 10:07:29.198103 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1912,"mutex_wait_us":26,"thread_start_us":1078,"threads_started":1}
I20260924 10:07:29.198494 7645 leader_election.cc:302] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fea1ff0f54e9b8220ae8f36c0006d, 9d02767a65b34c91a4bb29f5dfaa03de; no voters:
I20260924 10:07:29.198729 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.190304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:34857 (local address 127.6.2.194:43875)
0924 10:07:29.190480 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.190485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.191646 (+ 1161us) server_negotiation.cc:407] Connection header received
0924 10:07:29.191913 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.191918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.191995 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.192104 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:29.192572 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.193418 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.195892 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.197231 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.197545 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.197648 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.198071 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.198214 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:07:29.199261 7788 raft_consensus.cc:2781] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:07:29.199393 7788 raft_consensus.cc:482] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:07:29.199493 7788 raft_consensus.cc:3037] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:29.199653 7574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d38647a234c4f0a933f462d76f18ad7" candidate_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a476336c7544b45af7c4afa944b6388" is_pre_election: true
I20260924 10:07:29.200141 7574 raft_consensus.cc:2454] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d02767a65b34c91a4bb29f5dfaa03de in term 0.
I20260924 10:07:29.202198 7788 raft_consensus.cc:504] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.202800 7788 leader_election.cc:288] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [CANDIDATE]: Term 1 election: Requested vote from peers 6e3fea1ff0f54e9b8220ae8f36c0006d (127.6.2.193:35093), 9a476336c7544b45af7c4afa944b6388 (127.6.2.194:43875)
I20260924 10:07:29.203303 7439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d38647a234c4f0a933f462d76f18ad7" candidate_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d"
I20260924 10:07:29.203321 7574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d38647a234c4f0a933f462d76f18ad7" candidate_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a476336c7544b45af7c4afa944b6388"
I20260924 10:07:29.203513 7439 raft_consensus.cc:3037] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:29.203521 7574 raft_consensus.cc:3037] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:29.206516 7439 raft_consensus.cc:2454] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d02767a65b34c91a4bb29f5dfaa03de in term 1.
I20260924 10:07:29.206744 7574 raft_consensus.cc:2454] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d02767a65b34c91a4bb29f5dfaa03de in term 1.
I20260924 10:07:29.206923 7645 leader_election.cc:302] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fea1ff0f54e9b8220ae8f36c0006d, 9d02767a65b34c91a4bb29f5dfaa03de; no voters:
I20260924 10:07:29.207257 7788 raft_consensus.cc:2781] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:29.207724 7788 raft_consensus.cc:686] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [term 1 LEADER]: Becoming Leader. State: Replica: 9d02767a65b34c91a4bb29f5dfaa03de, State: Running, Role: LEADER
I20260924 10:07:29.208120 7788 consensus_queue.cc:237] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.211591 7276 catalog_manager.cc:5495] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d02767a65b34c91a4bb29f5dfaa03de (127.6.2.195). New cstate: current_term: 1 leader_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } health_report { overall_health: HEALTHY } } }
Sep 24 10:07:29 dist-test-slave-s3n3 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244446, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:29.255662 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.241415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.6.2.254:36433)
0924 10:07:29.242700 (+ 1285us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.242704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.242720 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:07:29.242758 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.242761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.242809 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.242926 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:29.244023 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.244810 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.246139 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.246538 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.246695 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.250397 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:29.250402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:29.250406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:29.250433 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:29.253019 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:29.253689 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:29.253697 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:29.253702 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:29.253818 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:29.254195 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:29.254200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:29.254203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:29.254448 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:29.254597 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.254972 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.255226 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1197,"thread_start_us":1098,"threads_started":1}
I20260924 10:07:29.255754 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.241356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36433 (local address 127.0.0.1:51506)
0924 10:07:29.241984 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:29.242013 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:29.242197 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:29.242950 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:29.242954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:29.243659 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:29.243853 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.243861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.244944 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.244951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.246012 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.246023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.246700 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.246706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.246936 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.247682 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:29.247698 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:29.250140 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:29.253161 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:29.253172 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:29.253181 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:29.253573 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:29.253922 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:29.253928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:29.253931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:29.254081 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:29.254573 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:29.254580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:29.254728 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 10:07:29.254901 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 10:07:29.255328 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":224,"thread_start_us":325,"threads_started":1}
I20260924 10:07:29.261751 7276 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:51506:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:07:29.262843 7276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:07:29.270045 7554 tablet_service.cc:1467] Processing CreateTablet for tablet dc6a742ddd2c4635a4d6b55f54df3c8a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd50adca3b584ecb876a1847d6137404]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:07:29.270037 7415 tablet_service.cc:1467] Processing CreateTablet for tablet dc6a742ddd2c4635a4d6b55f54df3c8a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd50adca3b584ecb876a1847d6137404]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:07:29.270630 7415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc6a742ddd2c4635a4d6b55f54df3c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:29.270635 7554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc6a742ddd2c4635a4d6b55f54df3c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:29.271593 7693 tablet_service.cc:1467] Processing CreateTablet for tablet dc6a742ddd2c4635a4d6b55f54df3c8a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd50adca3b584ecb876a1847d6137404]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:07:29.272114 7693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc6a742ddd2c4635a4d6b55f54df3c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:29.278019 7783 tablet_bootstrap.cc:492] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388: Bootstrap starting.
I20260924 10:07:29.278118 7785 tablet_bootstrap.cc:492] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d: Bootstrap starting.
I20260924 10:07:29.279811 7785 tablet_bootstrap.cc:654] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:29.279824 7783 tablet_bootstrap.cc:654] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:29.282066 7782 tablet_bootstrap.cc:492] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de: Bootstrap starting.
I20260924 10:07:29.283799 7782 tablet_bootstrap.cc:654] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:29.291908 7785 tablet_bootstrap.cc:492] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d: No bootstrap required, opened a new log
I20260924 10:07:29.292097 7785 ts_tablet_manager.cc:1397] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260924 10:07:29.292701 7783 tablet_bootstrap.cc:492] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388: No bootstrap required, opened a new log
I20260924 10:07:29.292886 7783 ts_tablet_manager.cc:1397] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.010s
I20260924 10:07:29.292862 7785 raft_consensus.cc:348] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.293125 7785 raft_consensus.cc:374] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:29.293211 7785 raft_consensus.cc:729] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3fea1ff0f54e9b8220ae8f36c0006d, State: Initialized, Role: FOLLOWER
I20260924 10:07:29.293496 7785 consensus_queue.cc:260] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.293829 7783 raft_consensus.cc:348] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.294092 7783 raft_consensus.cc:374] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:29.294188 7783 raft_consensus.cc:729] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a476336c7544b45af7c4afa944b6388, State: Initialized, Role: FOLLOWER
I20260924 10:07:29.293972 7790 tablet_replica.cc:439] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } }
I20260924 10:07:29.294364 7790 tablet_replica.cc:442] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:29.294504 7783 consensus_queue.cc:260] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.294806 7785 ts_tablet_manager.cc:1428] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 10:07:29.295014 7789 tablet_replica.cc:439] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } }
I20260924 10:07:29.295369 7789 tablet_replica.cc:442] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:29.296453 7783 ts_tablet_manager.cc:1428] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 10:07:29.299147 7782 tablet_bootstrap.cc:492] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de: No bootstrap required, opened a new log
I20260924 10:07:29.299412 7782 ts_tablet_manager.cc:1397] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260924 10:07:29.300184 7782 raft_consensus.cc:348] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.300509 7782 raft_consensus.cc:374] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:29.300598 7782 raft_consensus.cc:729] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d02767a65b34c91a4bb29f5dfaa03de, State: Initialized, Role: FOLLOWER
I20260924 10:07:29.300900 7782 consensus_queue.cc:260] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.301666 7782 ts_tablet_manager.cc:1428] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 10:07:29.301466 7788 tablet_replica.cc:439] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } }
I20260924 10:07:29.301851 7788 tablet_replica.cc:442] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:29.406098 7790 raft_consensus.cc:482] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:07:29.406359 7790 raft_consensus.cc:504] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.408397 7790 leader_election.cc:288] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a476336c7544b45af7c4afa944b6388 (127.6.2.194:43875), 9d02767a65b34c91a4bb29f5dfaa03de (127.6.2.195:41071)
I20260924 10:07:29.416352 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.408881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:45677 (local address 127.6.2.195:41071)
0924 10:07:29.409138 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.409147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.410086 (+ 939us) server_negotiation.cc:407] Connection header received
0924 10:07:29.410306 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.410311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.410400 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.410486 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:29.411345 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.412443 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.414418 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.415327 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.415464 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.415569 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.415912 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.416037 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 10:07:29.416419 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.408611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:41071 (local address 127.6.2.193:45677)
0924 10:07:29.409998 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:29.410021 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:29.410166 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:29.410544 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:29.410549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:29.410578 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:29.411143 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.411157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.412571 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.412579 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.414262 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.414274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.415538 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.415543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.415751 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.415773 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:07:29.415878 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 10:07:29.416007 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1278,"mutex_wait_us":649,"thread_start_us":462,"threads_started":1}
I20260924 10:07:29.417253 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc6a742ddd2c4635a4d6b55f54df3c8a" candidate_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" is_pre_election: true
I20260924 10:07:29.417542 7713 raft_consensus.cc:2454] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3fea1ff0f54e9b8220ae8f36c0006d in term 0.
I20260924 10:07:29.417632 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.408895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:35441 (local address 127.6.2.194:43875)
0924 10:07:29.409105 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.409113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.409818 (+ 705us) server_negotiation.cc:407] Connection header received
0924 10:07:29.410108 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.410115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.410237 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.410343 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:29.411019 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.412284 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.415082 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.416425 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.416567 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.416672 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.417224 (+ 552us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.417364 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 10:07:29.417804 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.408611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:43875 (local address 127.6.2.193:35441)
0924 10:07:29.409687 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:29.409713 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:29.409934 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:29.410383 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:29.410391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:29.410413 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:29.410868 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.410878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.412433 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.412439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.414842 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.414859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.416857 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.416863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.417033 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.417065 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:07:29.417178 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 10:07:29.417315 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":84,"thread_start_us":757,"threads_started":1}
I20260924 10:07:29.418393 7365 leader_election.cc:302] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fea1ff0f54e9b8220ae8f36c0006d, 9d02767a65b34c91a4bb29f5dfaa03de; no voters:
I20260924 10:07:29.418869 7790 raft_consensus.cc:2781] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:07:29.419024 7790 raft_consensus.cc:482] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:07:29.419212 7790 raft_consensus.cc:3037] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:29.419149 7574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc6a742ddd2c4635a4d6b55f54df3c8a" candidate_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a476336c7544b45af7c4afa944b6388" is_pre_election: true
I20260924 10:07:29.419409 7574 raft_consensus.cc:2454] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3fea1ff0f54e9b8220ae8f36c0006d in term 0.
I20260924 10:07:29.421800 7790 raft_consensus.cc:504] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.422782 7790 leader_election.cc:288] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [CANDIDATE]: Term 1 election: Requested vote from peers 9a476336c7544b45af7c4afa944b6388 (127.6.2.194:43875), 9d02767a65b34c91a4bb29f5dfaa03de (127.6.2.195:41071)
I20260924 10:07:29.423255 7574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc6a742ddd2c4635a4d6b55f54df3c8a" candidate_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a476336c7544b45af7c4afa944b6388"
I20260924 10:07:29.423316 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc6a742ddd2c4635a4d6b55f54df3c8a" candidate_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d02767a65b34c91a4bb29f5dfaa03de"
I20260924 10:07:29.423486 7574 raft_consensus.cc:3037] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:29.423527 7713 raft_consensus.cc:3037] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:29.425634 7574 raft_consensus.cc:2454] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3fea1ff0f54e9b8220ae8f36c0006d in term 1.
I20260924 10:07:29.426061 7367 leader_election.cc:302] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fea1ff0f54e9b8220ae8f36c0006d, 9a476336c7544b45af7c4afa944b6388; no voters:
I20260924 10:07:29.426357 7790 raft_consensus.cc:2781] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:29.426473 7713 raft_consensus.cc:2454] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3fea1ff0f54e9b8220ae8f36c0006d in term 1.
I20260924 10:07:29.426877 7790 raft_consensus.cc:686] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 1 LEADER]: Becoming Leader. State: Replica: 6e3fea1ff0f54e9b8220ae8f36c0006d, State: Running, Role: LEADER
I20260924 10:07:29.427512 7790 consensus_queue.cc:237] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } }
I20260924 10:07:29.429589 7828 tablet_replica.cc:439] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d: TxnStatusTablet state changed. Reason: New leader 6e3fea1ff0f54e9b8220ae8f36c0006d. Latest consensus state: current_term: 1 leader_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } }
I20260924 10:07:29.430027 7828 tablet_replica.cc:442] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:07:29.431129 7830 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:07:29.431567 7830 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:07:29.431476 7275 catalog_manager.cc:5495] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e3fea1ff0f54e9b8220ae8f36c0006d (127.6.2.193). New cstate: current_term: 1 leader_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } health_report { overall_health: UNKNOWN } } }
I20260924 10:07:29.485707 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.478039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:35093 (local address 127.0.0.1:48338)
0924 10:07:29.478392 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:29.478413 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:29.478541 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:29.479272 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:29.479278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:29.479292 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:07:29.479484 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.479495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.481069 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.481075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.482716 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.482731 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.483756 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.483763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.483986 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.484069 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:07:29.485086 (+ 1017us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:07:29.485109 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:07:29.485205 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 10:07:29.485382 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260924 10:07:29.486305 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.478059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.6.2.193:35093)
0924 10:07:29.478822 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.478829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.478846 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:07:29.478938 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.478943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.479046 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.479212 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:29.479652 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.480902 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.482894 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.483584 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.483729 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.484283 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:29.484964 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:29.485106 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.485936 (+ 830us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.486021 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624}
I20260924 10:07:29.496456 7574 raft_consensus.cc:1261] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388 [term 1 FOLLOWER]: Refusing update from remote peer 6e3fea1ff0f54e9b8220ae8f36c0006d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:29.497233 7713 raft_consensus.cc:1261] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de [term 1 FOLLOWER]: Refusing update from remote peer 6e3fea1ff0f54e9b8220ae8f36c0006d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:29.497372 7828 consensus_queue.cc:1035] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:29.497836 7790 consensus_queue.cc:1035] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:29.501183 7788 tablet_replica.cc:439] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de: TxnStatusTablet state changed. Reason: New leader 6e3fea1ff0f54e9b8220ae8f36c0006d. Latest consensus state: current_term: 1 leader_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } }
I20260924 10:07:29.501479 7788 tablet_replica.cc:442] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:29.507791 7828 tablet_replica.cc:439] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } }
I20260924 10:07:29.508145 7828 tablet_replica.cc:442] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:07:29.510799 7788 tablet_replica.cc:439] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } }
I20260924 10:07:29.511235 7788 tablet_replica.cc:442] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9d02767a65b34c91a4bb29f5dfaa03de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:29.515733 7789 tablet_replica.cc:439] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388: TxnStatusTablet state changed. Reason: New leader 6e3fea1ff0f54e9b8220ae8f36c0006d. Latest consensus state: current_term: 1 leader_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } }
I20260924 10:07:29.516026 7789 tablet_replica.cc:442] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:29.516820 7790 tablet_replica.cc:439] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } }
I20260924 10:07:29.517122 7790 tablet_replica.cc:442] T dc6a742ddd2c4635a4d6b55f54df3c8a P 6e3fea1ff0f54e9b8220ae8f36c0006d: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:07:29.520123 7789 tablet_replica.cc:439] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 } } peers { permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 } } peers { permanent_uuid: "9d02767a65b34c91a4bb29f5dfaa03de" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41071 } } }
I20260924 10:07:29.520375 7789 tablet_replica.cc:442] T dc6a742ddd2c4635a4d6b55f54df3c8a P 9a476336c7544b45af7c4afa944b6388: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:29.529112 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.522409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43182 (local address 127.6.2.195:41071)
0924 10:07:29.522584 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.522590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.523348 (+ 758us) server_negotiation.cc:407] Connection header received
0924 10:07:29.523522 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.523528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.523631 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.523791 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:29.524152 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.525281 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.526587 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.527314 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.527438 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.527811 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:29.528541 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:29.528692 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.528743 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.528829 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:07:29.545318 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.539210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:35093 (local address 127.6.2.195:34303)
0924 10:07:29.539853 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:29.539869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:29.539984 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:29.540329 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:29.540332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:29.540341 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:07:29.540620 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.540626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.541999 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.542003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.543151 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.543162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.544209 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.544213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.544326 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.544380 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:07:29.544796 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:07:29.544805 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 10:07:29.544873 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 10:07:29.544982 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":384,"threads_started":1}
I20260924 10:07:29.545385 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.539231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:34303 (local address 127.6.2.193:35093)
0924 10:07:29.539445 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.539452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.539959 (+ 507us) server_negotiation.cc:407] Connection header received
0924 10:07:29.540111 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.540117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.540204 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.540316 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:29.540755 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.541865 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.543286 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.544034 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.544167 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.544538 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:29.544686 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:29.544779 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.544926 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.545012 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:07:29.555461 7574 raft_consensus.cc:1261] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388 [term 1 FOLLOWER]: Refusing update from remote peer 9d02767a65b34c91a4bb29f5dfaa03de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:29.555719 7439 raft_consensus.cc:1261] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 1 FOLLOWER]: Refusing update from remote peer 9d02767a65b34c91a4bb29f5dfaa03de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:29.556137 7814 consensus_queue.cc:1035] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a476336c7544b45af7c4afa944b6388" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 43875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:29.556523 7788 consensus_queue.cc:1035] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3fea1ff0f54e9b8220ae8f36c0006d" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:29.560559 7834 mvcc.cc:205] Tried to move back new op lower bound from 7332841265373327360 to 7332841263960690688. Current Snapshot: MvccSnapshot[applied={T|T < 7332841265373327360}]
I20260924 10:07:29.596968 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.589757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:58151 (local address 127.6.2.195:41071)
0924 10:07:29.589932 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.589938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.590319 (+ 381us) server_negotiation.cc:407] Connection header received
0924 10:07:29.590590 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.590595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.590679 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.590744 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:29.591777 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.592850 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.594590 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.595411 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.595543 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.596140 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:29.596301 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:29.596367 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.596609 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.596712 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:07:29.597750 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.589612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:41071 (local address 127.6.2.193:58151)
0924 10:07:29.590216 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:29.590239 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:29.590431 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:29.591165 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:29.591170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:29.591181 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:07:29.591599 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.591610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.592997 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.593001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.594453 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:29.594465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.595614 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.595622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:29.595854 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.595931 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:07:29.596464 (+ 533us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:07:29.596481 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:07:29.597156 (+ 675us) client_negotiation.cc:240] Negotiation successful
0924 10:07:29.597313 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"spinlock_wait_cycles":2304,"thread_start_us":392,"threads_started":1}
I20260924 10:07:29.661010 7276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:29.661294 7276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:29.664496 7276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00d10669f9904852a09828c543c448be: Sending DeleteTablet for 3 replicas of tablet 8d38647a234c4f0a933f462d76f18ad7
I20260924 10:07:29.665277 7415 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d38647a234c4f0a933f462d76f18ad7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:29 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:48332
I20260924 10:07:29.665956 7554 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d38647a234c4f0a933f462d76f18ad7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:29 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:56858
I20260924 10:07:29.672063 7693 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d38647a234c4f0a933f462d76f18ad7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:29 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:43176
I20260924 10:07:29.687871 7858 tablet_replica.cc:330] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de: stopping tablet replica
I20260924 10:07:29.688177 7858 raft_consensus.cc:2229] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:07:29.689132 7858 raft_consensus.cc:2258] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:29.690766 7858 ts_tablet_manager.cc:1905] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:29.696002 7855 tablet_replica.cc:330] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d: stopping tablet replica
I20260924 10:07:29.696249 7855 raft_consensus.cc:2229] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:29.696559 7855 raft_consensus.cc:2258] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:29.696816 7856 tablet_replica.cc:330] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388: stopping tablet replica
I20260924 10:07:29.697039 7856 raft_consensus.cc:2229] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:29.697291 7856 raft_consensus.cc:2258] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:29.698107 7855 ts_tablet_manager.cc:1905] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:29.699836 7856 ts_tablet_manager.cc:1905] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:29.705452 7855 ts_tablet_manager.cc:1918] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:07:29.705598 7855 log.cc:1196] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/8d38647a234c4f0a933f462d76f18ad7
I20260924 10:07:29.706251 7855 ts_tablet_manager.cc:1939] T 8d38647a234c4f0a933f462d76f18ad7 P 6e3fea1ff0f54e9b8220ae8f36c0006d: Deleting consensus metadata
I20260924 10:07:29.707542 7263 catalog_manager.cc:4841] TS 6e3fea1ff0f54e9b8220ae8f36c0006d (127.6.2.193:35093): tablet 8d38647a234c4f0a933f462d76f18ad7 (table test-table [id=cc8b1e3431f340c084214b41da6fa276]) successfully deleted
I20260924 10:07:29.710052 7858 ts_tablet_manager.cc:1918] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:07:29.710239 7858 log.cc:1196] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/8d38647a234c4f0a933f462d76f18ad7
I20260924 10:07:29.710775 7858 ts_tablet_manager.cc:1939] T 8d38647a234c4f0a933f462d76f18ad7 P 9d02767a65b34c91a4bb29f5dfaa03de: Deleting consensus metadata
I20260924 10:07:29.712003 7262 catalog_manager.cc:4841] TS 9d02767a65b34c91a4bb29f5dfaa03de (127.6.2.195:41071): tablet 8d38647a234c4f0a933f462d76f18ad7 (table test-table [id=cc8b1e3431f340c084214b41da6fa276]) successfully deleted
I20260924 10:07:29.712553 7856 ts_tablet_manager.cc:1918] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:07:29.712723 7856 log.cc:1196] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/8d38647a234c4f0a933f462d76f18ad7
I20260924 10:07:29.713179 7856 ts_tablet_manager.cc:1939] T 8d38647a234c4f0a933f462d76f18ad7 P 9a476336c7544b45af7c4afa944b6388: Deleting consensus metadata
I20260924 10:07:29.714175 7264 catalog_manager.cc:4841] TS 9a476336c7544b45af7c4afa944b6388 (127.6.2.194:43875): tablet 8d38647a234c4f0a933f462d76f18ad7 (table test-table [id=cc8b1e3431f340c084214b41da6fa276]) successfully deleted
Sep 24 10:07:29 dist-test-slave-s3n3 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:07:29.729383 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.684709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.6.2.193:35093)
0924 10:07:29.684924 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.684930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.700228 (+ 15298us) server_negotiation.cc:407] Connection header received
0924 10:07:29.711455 (+ 11227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.711461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.711555 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.711628 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:29.713824 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.714896 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.716610 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.717073 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.717260 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.722615 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:29.722628 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:29.722644 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:29.722685 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:29.726251 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:29.727106 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:29.727117 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:29.727121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:29.727267 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:29.727713 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:29.727724 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:29.727726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:29.728093 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:29.728452 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.728758 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.729000 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260924 10:07:29.730230 7480 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48352
I20260924 10:07:29.749256 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.733615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.6.2.254:36433)
0924 10:07:29.733828 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.733835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.735593 (+ 1758us) server_negotiation.cc:407] Connection header received
0924 10:07:29.735807 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.735811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.735873 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.736011 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:29.737442 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.738227 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.739548 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.739979 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.740112 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.742773 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:29.742778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:29.742782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:29.742816 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:29.745962 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:29.746692 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:29.746698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:29.746704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:29.746851 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:29.747299 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:29.747307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:29.747310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:29.747641 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:29.747999 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.748474 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.748765 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 10:07:29.749872 7276 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51512
I20260924 10:07:29.752030 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 7349
I20260924 10:07:29.767201 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.753841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.6.2.193:35093)
0924 10:07:29.754267 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.754275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.754339 (+ 64us) server_negotiation.cc:407] Connection header received
0924 10:07:29.754626 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.754635 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.754723 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.754805 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:29.756137 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.756937 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.758150 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.758682 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.758825 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.761095 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:29.761109 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:29.761114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:29.761154 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:29.763970 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:29.764621 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:29.764627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:29.764636 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:29.764762 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:29.765134 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:29.765143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:29.765146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:29.765455 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:29.765774 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.766575 (+ 801us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.766828 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306}
W20260924 10:07:29.767786 7480 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48358
W20260924 10:07:29.768342 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 7349: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:07:29.768524 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 7349
I20260924 10:07:29.789520 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 7489
Sep 24 10:07:29 dist-test-slave-s3n3 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:07:29.806887 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.790369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.6.2.194:43875)
0924 10:07:29.790591 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.790599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.790620 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:07:29.790849 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.790855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.790935 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.791064 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:29.792332 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.793114 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.794453 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.794977 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.795265 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.800031 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:29.800052 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:29.800075 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:29.800121 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:29.804010 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:29.804750 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:29.804760 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:29.804764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:29.804919 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:29.805335 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:29.805340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:29.805344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:29.805785 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:29.806104 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.806270 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.806498 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260924 10:07:29.807749 7619 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56880
W20260924 10:07:29.808461 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 7489: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:07:29.808642 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 7489
I20260924 10:07:29.830654 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 7628
Sep 24 10:07:29 dist-test-slave-s3n3 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:07:29.848222 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:29.831468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.6.2.195:41071)
0924 10:07:29.831651 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:29.831657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:29.831678 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:07:29.831836 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:29.831841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:29.831929 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:29.832055 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:29.833198 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.834342 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.835619 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:29.836295 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:29.836430 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:29.840860 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:29.840880 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:29.840904 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:29.840954 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:29.844846 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:29.845644 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:29.845653 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:29.845658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:29.845850 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:29.846413 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:29.846418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:29.846421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:29.846922 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:29.847331 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:29.847548 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 10:07:29.847796 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 10:07:29.849041 7758 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43190
W20260924 10:07:29.849774 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 7628: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:07:29.849938 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 7628
I20260924 10:07:29.865092 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 7246
I20260924 10:07:29.866252 7318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:29.981900 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 7246
2026-09-24T10:07:30Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4712 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:07:30 dist-test-slave-s3n3 krb5kdc[7866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:07:30 dist-test-slave-s3n3 krb5kdc[7866](info): set up 2 sockets
Sep 24 10:07:30 dist-test-slave-s3n3 krb5kdc[7866](info): commencing operation
krb5kdc: starting...
W20260924 10:07:31.087689 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:07:31 dist-test-slave-s3n3 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:07:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:07:31.283560 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:35989
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:31.602475 7879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:31.602757 7879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:31.602921 7879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:31.610077 7879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:31.610181 7879 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:07:31.610314 7879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:31.610433 7879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:31.610517 7879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:31.610634 7879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:31.621097 7879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:35989
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:35989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:31.622479 7879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:31.623823 7879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:31.635711 7887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:31.635736 7884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:31.635855 7885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:31.636600 7879 server_base.cc:1034] running on GCE node
I20260924 10:07:31.637459 7879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:31.638864 7879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:31.640132 7879 hybrid_clock.cc:648] HybridClock initialized: now 1790244451640053 us; error 61 us; skew 500 ppm
I20260924 10:07:31.640522 7879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:31 dist-test-slave-s3n3 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244451, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:31.643723 7879 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:07:31.645210 7879 webserver.cc:460] Webserver started at http://127.6.2.254:43793/ using document root <none> and password file <none>
I20260924 10:07:31.645910 7879 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:31.646044 7879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:31.646374 7879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:31.649390 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "844ec8e9f4db407fb1077dfefdd615fc"
format_stamp: "Formatted at 2026-09-24 10:07:31 on dist-test-slave-s3n3"
server_key: "eeafd9041629b544ff30a7334bbdc883"
server_key_iv: "be07eaa40bf8117f6fdae2670b4c9694"
server_key_version: "encryptionkey@0"
I20260924 10:07:31.650169 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "844ec8e9f4db407fb1077dfefdd615fc"
format_stamp: "Formatted at 2026-09-24 10:07:31 on dist-test-slave-s3n3"
server_key: "eeafd9041629b544ff30a7334bbdc883"
server_key_iv: "be07eaa40bf8117f6fdae2670b4c9694"
server_key_version: "encryptionkey@0"
I20260924 10:07:31.655912 7879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 10:07:31.659439 7894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:31.660782 7879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:07:31.660974 7879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "844ec8e9f4db407fb1077dfefdd615fc"
format_stamp: "Formatted at 2026-09-24 10:07:31 on dist-test-slave-s3n3"
server_key: "eeafd9041629b544ff30a7334bbdc883"
server_key_iv: "be07eaa40bf8117f6fdae2670b4c9694"
server_key_version: "encryptionkey@0"
I20260924 10:07:31.661212 7879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:31.672358 7879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:31.676461 7879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:31.676731 7879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:31.698998 7879 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:35989
I20260924 10:07:31.699016 7955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:35989 every 8 connection(s)
I20260924 10:07:31.700387 7879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:31.701557 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 7879
I20260924 10:07:31.701774 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:07:31.702571 6155 external_mini_cluster.cc:1428] Setting key c485f32e3c039f6ed51a8d196197e2a9
I20260924 10:07:31.705109 7956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 10:07:31 dist-test-slave-s3n3 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:31.718215 7956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc: Bootstrap starting.
I20260924 10:07:31.722383 7956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:31.723706 7956 log.cc:824] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:31.726471 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:31.704745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.6.2.254:35989)
0924 10:07:31.705556 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:31.705567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:31.705605 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:07:31.707244 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:31.707264 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:31.707639 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:31.707978 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:31.709708 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:31.711042 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:31.712770 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:31.713398 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:31.713527 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:31.718151 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:31.718161 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:31.718180 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:31.718232 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:31.721960 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:31.722744 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:31.722758 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:31.722769 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:31.722985 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:31.723416 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:31.723420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:31.723423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:31.724095 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:31.724447 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:31.724889 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 10:07:31.725239 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":411,"threads_started":1}
I20260924 10:07:31.727137 7956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc: No bootstrap required, opened a new log
I20260924 10:07:31.735044 7956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "844ec8e9f4db407fb1077dfefdd615fc" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35989 } }
I20260924 10:07:31.735469 7956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:31.735543 7956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844ec8e9f4db407fb1077dfefdd615fc, State: Initialized, Role: FOLLOWER
I20260924 10:07:31.736025 7956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "844ec8e9f4db407fb1077dfefdd615fc" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35989 } }
I20260924 10:07:31.736208 7956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:31.736294 7956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:31.736415 7956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:31.738523 7956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "844ec8e9f4db407fb1077dfefdd615fc" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35989 } }
I20260924 10:07:31.739002 7956 leader_election.cc:302] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 844ec8e9f4db407fb1077dfefdd615fc; no voters:
I20260924 10:07:31.739753 7956 leader_election.cc:288] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:31.739897 7961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:31.740582 7961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [term 1 LEADER]: Becoming Leader. State: Replica: 844ec8e9f4db407fb1077dfefdd615fc, State: Running, Role: LEADER
I20260924 10:07:31.741096 7961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "844ec8e9f4db407fb1077dfefdd615fc" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35989 } }
I20260924 10:07:31.741909 7956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:31.744216 7963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 844ec8e9f4db407fb1077dfefdd615fc. Latest consensus state: current_term: 1 leader_uuid: "844ec8e9f4db407fb1077dfefdd615fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "844ec8e9f4db407fb1077dfefdd615fc" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35989 } } }
I20260924 10:07:31.744176 7962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "844ec8e9f4db407fb1077dfefdd615fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "844ec8e9f4db407fb1077dfefdd615fc" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35989 } } }
I20260924 10:07:31.744680 7963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:31.744738 7962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:31.751185 7967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:31.757078 7967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:31.765762 7967 catalog_manager.cc:1347] Generated new cluster ID: 9d08ea05c6714b22a596f717f6decacf
I20260924 10:07:31.765887 7967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:31.782157 7967 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:31.783740 7967 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:31.793612 7967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc: Generated new TSK 0
I20260924 10:07:31.794539 7967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:07:31.855340 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:35989
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:32.193233 7982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:32.193560 7982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:32.193773 7982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:32.202966 7982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:32.203133 7982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:32.203286 7982 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:07:32.203372 7982 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:07:32.203444 7982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:07:32.213385 7982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.2.254:35989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:32.214691 7982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:32.216001 7982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:32.229046 7988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:32.229159 7990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:32.229226 7987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:32.230221 7982 server_base.cc:1034] running on GCE node
I20260924 10:07:32.230852 7982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:32.231909 7982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:32.233139 7982 hybrid_clock.cc:648] HybridClock initialized: now 1790244452233090 us; error 44 us; skew 500 ppm
I20260924 10:07:32.233479 7982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:32 dist-test-slave-s3n3 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244452, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:32.236637 7982 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:07:32.238092 7982 webserver.cc:460] Webserver started at http://127.6.2.193:45989/ using document root <none> and password file <none>
I20260924 10:07:32.238785 7982 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:32.238927 7982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:32.239285 7982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:32.242224 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "97ce72fd9b574bdb9a37dd06cb75bace"
format_stamp: "Formatted at 2026-09-24 10:07:32 on dist-test-slave-s3n3"
server_key: "9669c01bcde8ab69294a2773b780656e"
server_key_iv: "eb6c580fb1f8d71ec4605cda89848909"
server_key_version: "encryptionkey@0"
I20260924 10:07:32.243105 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "97ce72fd9b574bdb9a37dd06cb75bace"
format_stamp: "Formatted at 2026-09-24 10:07:32 on dist-test-slave-s3n3"
server_key: "9669c01bcde8ab69294a2773b780656e"
server_key_iv: "eb6c580fb1f8d71ec4605cda89848909"
server_key_version: "encryptionkey@0"
I20260924 10:07:32.248131 7982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:07:32.251632 7997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:32.253032 7982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:07:32.253201 7982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "97ce72fd9b574bdb9a37dd06cb75bace"
format_stamp: "Formatted at 2026-09-24 10:07:32 on dist-test-slave-s3n3"
server_key: "9669c01bcde8ab69294a2773b780656e"
server_key_iv: "eb6c580fb1f8d71ec4605cda89848909"
server_key_version: "encryptionkey@0"
I20260924 10:07:32.253383 7982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:32.273905 7982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:32.277755 7982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:32.277997 7982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:32.280544 7982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:32.280635 7982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:32.280822 7982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:32.280908 7982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:07:32 dist-test-slave-s3n3 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244452, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:32.303468 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:32.281016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:37829 (local address 127.6.2.254:35989)
0924 10:07:32.281769 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:32.281776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:32.282776 (+ 1000us) server_negotiation.cc:407] Connection header received
0924 10:07:32.284933 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:32.284939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:32.285023 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:32.285160 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:32.287458 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.288768 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.290981 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.291891 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.292062 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:32.295694 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:32.295701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:32.295706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:32.295742 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:32.299335 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:32.300357 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:32.300365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:32.300370 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:32.300536 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:32.301253 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:32.301259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:32.301263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:32.301698 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:32.301862 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:32.302607 (+ 745us) server_negotiation.cc:299] Negotiation successful
0924 10:07:32.302903 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":527,"threads_started":1}
I20260924 10:07:32.304068 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:32.281703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:35989 (local address 127.6.2.193:37829)
0924 10:07:32.282587 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:32.282634 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:32.284606 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:32.285415 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:32.285431 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:32.286383 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:32.287232 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:32.287278 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.288994 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.289002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:32.290829 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:32.290841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.292124 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.292129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:32.292297 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:32.292995 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:32.293021 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:32.295402 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:32.299476 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:32.299489 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:32.299506 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:32.300205 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:32.300699 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:32.300707 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:32.300711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:32.301078 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:32.301859 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:32.301868 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:32.302204 (+ 336us) client_negotiation.cc:769] Sending connection context
0924 10:07:32.302438 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 10:07:32.302811 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":398,"threads_started":1}
I20260924 10:07:32.327919 8117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:39011 every 8 connection(s)
I20260924 10:07:32.327926 7982 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:39011
I20260924 10:07:32.328858 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:32.311175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:44783 (local address 127.6.2.254:35989)
0924 10:07:32.311379 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:32.311386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:32.312287 (+ 901us) server_negotiation.cc:407] Connection header received
0924 10:07:32.312506 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:32.312512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:32.312605 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:32.312733 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:32.314262 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.315550 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.318201 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.318801 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.318942 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:32.321807 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:32.321814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:32.321822 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:32.321869 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:32.325702 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:32.326589 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:32.326593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:32.326596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:32.326778 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:32.327298 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:32.327303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:32.327307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:32.327693 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:32.327840 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:32.328458 (+ 618us) server_negotiation.cc:299] Negotiation successful
0924 10:07:32.328621 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:07:32.329105 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:32.311273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:35989 (local address 127.6.2.193:44783)
0924 10:07:32.312127 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:32.312173 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:32.312355 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:32.312763 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:32.312771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:32.313575 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:32.314072 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:32.314090 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.315740 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.315749 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:32.318034 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:32.318053 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.319267 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.319273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:32.319514 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:32.320194 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:32.320222 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:32.321534 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:32.325910 (+ 4376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:32.325916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:32.325924 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:32.326454 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:32.326916 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:32.326921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:32.326924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:32.327129 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:32.327952 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:32.327957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:32.328193 (+ 236us) client_negotiation.cc:769] Sending connection context
0924 10:07:32.328341 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 10:07:32.328667 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":498,"threads_started":1}
I20260924 10:07:32.329635 7982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:07:32.331877 8118 heartbeater.cc:347] Connected to a master server at 127.6.2.254:35989
I20260924 10:07:32.332180 8118 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:32.333348 8118 heartbeater.cc:511] Master 127.6.2.254:35989 requested a full tablet report, sending...
I20260924 10:07:32.335686 7911 ts_manager.cc:194] Registered new tserver with Master: 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193:39011)
I20260924 10:07:32.335799 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 7982
I20260924 10:07:32.336048 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:07:32.336666 6155 external_mini_cluster.cc:1428] Setting key bc43ea31e7c2814303600d599daa4f44
I20260924 10:07:32.337253 7911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:37829
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:07:32.378432 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:35989
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:32.694844 8122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:32.695218 8122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:32.695390 8122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:32.705307 8122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:32.705456 8122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:32.705601 8122 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:07:32.705672 8122 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:07:32.705773 8122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:07:32.721498 8122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.2.254:35989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:32.722971 8122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:32.724318 8122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:32.737673 8127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:32.737710 8128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:32.737668 8130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:32.738953 8122 server_base.cc:1034] running on GCE node
I20260924 10:07:32.739593 8122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:32.740496 8122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:32.741735 8122 hybrid_clock.cc:648] HybridClock initialized: now 1790244452741656 us; error 66 us; skew 500 ppm
I20260924 10:07:32.742072 8122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:32 dist-test-slave-s3n3 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244452, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:32.745424 8122 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:07:32.746975 8122 webserver.cc:460] Webserver started at http://127.6.2.194:37811/ using document root <none> and password file <none>
I20260924 10:07:32.747713 8122 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:32.747817 8122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:32.748057 8122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:32.750878 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "151858e25ef04c8c8e0f37c93e1b7872"
format_stamp: "Formatted at 2026-09-24 10:07:32 on dist-test-slave-s3n3"
server_key: "0c2ddf6c55b58335ef1f4e3b99252a68"
server_key_iv: "4c5af8dce3bf46fc9946bba51486ede0"
server_key_version: "encryptionkey@0"
I20260924 10:07:32.751653 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "151858e25ef04c8c8e0f37c93e1b7872"
format_stamp: "Formatted at 2026-09-24 10:07:32 on dist-test-slave-s3n3"
server_key: "0c2ddf6c55b58335ef1f4e3b99252a68"
server_key_iv: "4c5af8dce3bf46fc9946bba51486ede0"
server_key_version: "encryptionkey@0"
I20260924 10:07:32.756829 8122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:07:32.760504 8137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:32.761925 8122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:32.762102 8122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "151858e25ef04c8c8e0f37c93e1b7872"
format_stamp: "Formatted at 2026-09-24 10:07:32 on dist-test-slave-s3n3"
server_key: "0c2ddf6c55b58335ef1f4e3b99252a68"
server_key_iv: "4c5af8dce3bf46fc9946bba51486ede0"
server_key_version: "encryptionkey@0"
I20260924 10:07:32.762323 8122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:32.787324 8122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:32.791720 8122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:32.792019 8122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:32.794895 8122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:32.795015 8122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:32.795168 8122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:32.795260 8122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:07:32 dist-test-slave-s3n3 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244452, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:32.818382 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:32.795348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:39423 (local address 127.6.2.254:35989)
0924 10:07:32.795532 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:32.795540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:32.796999 (+ 1459us) server_negotiation.cc:407] Connection header received
0924 10:07:32.799173 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:32.799179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:32.799278 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:32.799394 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:32.801516 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.802368 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.805123 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.805871 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.805988 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:32.811232 (+ 5244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:32.811242 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:32.811248 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:32.811310 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:32.814603 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:32.815663 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:32.815672 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:32.815678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:32.815865 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:32.816551 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:32.816557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:32.816563 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:32.816888 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:32.817058 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:32.817802 (+ 744us) server_negotiation.cc:299] Negotiation successful
0924 10:07:32.818014 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:07:32.819691 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:32.796059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:35989 (local address 127.6.2.194:39423)
0924 10:07:32.796839 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:32.796877 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:32.798846 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:32.799674 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:32.799688 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:32.800469 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:32.801325 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:32.801348 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.802538 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.802545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:32.804972 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:32.804990 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.806121 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.806128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:32.806362 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:32.807385 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:32.807428 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:32.810956 (+ 3528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:32.814766 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:32.814779 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:32.814798 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:32.815493 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:32.816036 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:32.816048 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:32.816054 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:32.816384 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:32.817025 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:32.817033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:32.817428 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 10:07:32.817700 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 10:07:32.818240 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":348,"threads_started":1}
I20260924 10:07:32.843628 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:32.827385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:60903 (local address 127.6.2.254:35989)
0924 10:07:32.827599 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:32.827606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:32.828500 (+ 894us) server_negotiation.cc:407] Connection header received
0924 10:07:32.828718 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:32.828727 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:32.828830 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:32.828951 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:32.830187 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.831696 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.833987 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.834418 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.834570 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:32.837004 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:32.837010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:32.837017 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:32.837052 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:32.840005 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:32.841102 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:32.841110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:32.841115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:32.841272 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:32.841746 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:32.841754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:32.841759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:32.842312 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:32.842498 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:32.842935 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 10:07:32.843206 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:07:32.843744 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:32.827434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:35989 (local address 127.6.2.194:60903)
0924 10:07:32.828360 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:32.828384 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:32.828561 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:32.828997 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:32.829005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:32.829723 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:32.830028 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:32.830037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.831907 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.831919 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:32.833819 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:32.833837 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:32.834613 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:32.834619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:32.834801 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:32.835544 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:32.835564 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:32.836818 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:32.840172 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:32.840178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:32.840182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:32.840967 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:32.841398 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:32.841403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:32.841407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:32.841611 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:32.842449 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:32.842457 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:32.842697 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 10:07:32.842815 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 10:07:32.843212 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":584,"threads_started":1}
I20260924 10:07:32.844884 8122 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:34683
I20260924 10:07:32.844901 8256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:34683 every 8 connection(s)
I20260924 10:07:32.846491 8122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:07:32.846850 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 8122
I20260924 10:07:32.847038 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:07:32.847692 6155 external_mini_cluster.cc:1428] Setting key 2607f5467f9fa91fc5356411b30f0042
I20260924 10:07:32.849021 8257 heartbeater.cc:347] Connected to a master server at 127.6.2.254:35989
I20260924 10:07:32.849264 8257 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:32.850008 8257 heartbeater.cc:511] Master 127.6.2.254:35989 requested a full tablet report, sending...
I20260924 10:07:32.851626 7911 ts_manager.cc:194] Registered new tserver with Master: 151858e25ef04c8c8e0f37c93e1b7872 (127.6.2.194:34683)
I20260924 10:07:32.852888 7911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:39423
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:07:32.890903 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:35989
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:33.203896 8261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:33.204164 8261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:33.204264 8261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:33.211470 8261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:33.211578 8261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:33.211663 8261 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:07:33.211722 8261 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:07:33.211789 8261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:07:33.221370 8261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.2.254:35989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:33.222666 8261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:33.223961 8261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:33.236747 8269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:33.236883 8266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:33.236853 8267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:33.238116 8261 server_base.cc:1034] running on GCE node
I20260924 10:07:33.238656 8261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:33.239612 8261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:33.240824 8261 hybrid_clock.cc:648] HybridClock initialized: now 1790244453240782 us; error 29 us; skew 500 ppm
I20260924 10:07:33.241154 8261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:33 dist-test-slave-s3n3 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244453, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:33.251289 8261 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:07:33.252903 8261 webserver.cc:460] Webserver started at http://127.6.2.195:43799/ using document root <none> and password file <none>
I20260924 10:07:33.253595 8261 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:33.253695 8261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:33.253957 8261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:33.256860 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "307d18d9f9bb4d19baa09b4db2090b70"
format_stamp: "Formatted at 2026-09-24 10:07:33 on dist-test-slave-s3n3"
server_key: "2ac663f328e9b2f9b79d7c2a56b41689"
server_key_iv: "46a6d2baca98ad64b569f6be57aa65b9"
server_key_version: "encryptionkey@0"
I20260924 10:07:33.257589 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "307d18d9f9bb4d19baa09b4db2090b70"
format_stamp: "Formatted at 2026-09-24 10:07:33 on dist-test-slave-s3n3"
server_key: "2ac663f328e9b2f9b79d7c2a56b41689"
server_key_iv: "46a6d2baca98ad64b569f6be57aa65b9"
server_key_version: "encryptionkey@0"
I20260924 10:07:33.262650 8261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:07:33.266234 8276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:33.267613 8261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:07:33.267788 8261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "307d18d9f9bb4d19baa09b4db2090b70"
format_stamp: "Formatted at 2026-09-24 10:07:33 on dist-test-slave-s3n3"
server_key: "2ac663f328e9b2f9b79d7c2a56b41689"
server_key_iv: "46a6d2baca98ad64b569f6be57aa65b9"
server_key_version: "encryptionkey@0"
I20260924 10:07:33.267980 8261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:33.289812 8261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:33.293716 8261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:33.294044 8261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:33.296748 8261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:33.296919 8261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:33.297082 8261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:33.297215 8261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:07:33 dist-test-slave-s3n3 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244453, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:33.319012 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.297316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:57779 (local address 127.6.2.254:35989)
0924 10:07:33.297533 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:33.297541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:33.298891 (+ 1350us) server_negotiation.cc:407] Connection header received
0924 10:07:33.300896 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:33.300902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:33.301006 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:33.301117 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:33.303147 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.304479 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.306786 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.307576 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.307692 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.312238 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:33.312244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:33.312249 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:33.312298 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:33.314919 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:33.315840 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:33.315846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:33.315850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:33.315979 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:33.316586 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:33.316594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:33.316598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:33.316993 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:33.317177 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:33.318170 (+ 993us) server_negotiation.cc:299] Negotiation successful
0924 10:07:33.318495 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:07:33.319686 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.297934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:35989 (local address 127.6.2.195:57779)
0924 10:07:33.298742 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:33.298777 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:33.300632 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:33.301346 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:33.301354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:33.302065 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:33.302910 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.302940 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.304664 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.304670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.306633 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.306644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.307803 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.307808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.307976 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.308738 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:33.308794 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:33.312018 (+ 3224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:33.315053 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:33.315084 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:33.315100 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:33.315667 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:33.316164 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:33.316170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:33.316175 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:33.316460 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:33.317158 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:33.317165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:33.317548 (+ 383us) client_negotiation.cc:769] Sending connection context
0924 10:07:33.317966 (+ 418us) client_negotiation.cc:240] Negotiation successful
0924 10:07:33.318490 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":378,"threads_started":1}
I20260924 10:07:33.339731 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.325114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:56941 (local address 127.6.2.254:35989)
0924 10:07:33.325330 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:33.325334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:33.326621 (+ 1287us) server_negotiation.cc:407] Connection header received
0924 10:07:33.326809 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:33.326817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:33.326927 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:33.327044 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:33.327903 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.328583 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.329790 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.330209 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.330344 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.332474 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:33.332482 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:33.332489 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:33.332532 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:33.336265 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:33.337189 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:33.337197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:33.337203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:33.337373 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:33.337830 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:33.337838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:33.337844 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:33.338247 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:33.338478 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:33.338990 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 10:07:33.339275 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:07:33.339947 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.325625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:35989 (local address 127.6.2.195:56941)
0924 10:07:33.326468 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:33.326485 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:33.326668 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:33.327119 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:33.327128 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:33.327523 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:33.327766 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.327774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.328723 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.328728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.329641 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.329651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.330382 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.330388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.330559 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.331252 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:33.331312 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:33.332238 (+ 926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:33.336404 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:33.336409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:33.336415 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:33.337048 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:33.337505 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:33.337511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:33.337517 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:33.337694 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:33.338405 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:33.338412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:33.338681 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 10:07:33.338872 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 10:07:33.339272 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":531,"threads_started":1}
I20260924 10:07:33.340729 8118 heartbeater.cc:503] Master 127.6.2.254:35989 was elected leader, sending a full tablet report...
I20260924 10:07:33.354450 8261 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:42961
I20260924 10:07:33.354483 8395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:42961 every 8 connection(s)
I20260924 10:07:33.355700 8261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:07:33.357856 8396 heartbeater.cc:347] Connected to a master server at 127.6.2.254:35989
I20260924 10:07:33.358034 8396 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:33.358592 8396 heartbeater.cc:511] Master 127.6.2.254:35989 requested a full tablet report, sending...
I20260924 10:07:33.358628 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 8261
I20260924 10:07:33.358788 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:07:33.359297 6155 external_mini_cluster.cc:1428] Setting key 00ec49d902c398d39db756007c9e3ca3
I20260924 10:07:33.360093 7911 ts_manager.cc:194] Registered new tserver with Master: 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195:42961)
I20260924 10:07:33.360980 7911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:57779
I20260924 10:07:33.362252 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:07:33 dist-test-slave-s3n3 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:07:33 dist-test-slave-s3n3 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:33.407025 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.393490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.6.2.254:35989)
0924 10:07:33.393659 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:33.393663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:33.393936 (+ 273us) server_negotiation.cc:407] Connection header received
0924 10:07:33.394078 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:33.394083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:33.394143 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:33.394236 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:33.395609 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.396357 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.397565 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.397990 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.398085 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.401858 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:33.401863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:33.401866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:33.401900 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:33.404501 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:33.405068 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:33.405072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:33.405075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:33.405240 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:33.405662 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:33.405666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:33.405668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:33.405981 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:33.406160 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:33.406524 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 10:07:33.406686 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:07:33.410946 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:07:33.417718 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:07:33.453047 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.429699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:34683 (local address 127.0.0.1:45864)
0924 10:07:33.431724 (+ 2025us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:33.431772 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:33.431936 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:33.442786 (+ 10850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:33.442798 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:33.442854 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:33.443626 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.443645 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.447435 (+ 3790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.447444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.450140 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.450155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.452149 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.452155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.452326 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.452356 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:07:33.452433 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:07:33.452576 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1693,"mutex_wait_us":7,"thread_start_us":1752,"threads_started":1}
I20260924 10:07:33.453545 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.430050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:39011 (local address 127.0.0.1:46564)
0924 10:07:33.443543 (+ 13493us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:33.443571 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:33.443722 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:33.444762 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:33.444770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:33.444801 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:33.445280 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.445305 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.447702 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.447710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.450333 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.450350 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.451823 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.451829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.452056 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.452117 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 10:07:33.452253 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 10:07:33.452445 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13301,"thread_start_us":13165,"threads_started":1}
I20260924 10:07:33.453778 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.430527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.6.2.193:39011)
0924 10:07:33.441278 (+ 10751us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:33.441300 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:33.443884 (+ 2584us) server_negotiation.cc:407] Connection header received
0924 10:07:33.444008 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:33.444019 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:33.444297 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:33.444506 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:33.445506 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.447500 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.450504 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.451639 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.451802 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.452016 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:33.452624 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 10:07:33.452850 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10536,"thread_start_us":10771,"threads_started":1}
I20260924 10:07:33.453856 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.430095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.6.2.194:34683)
0924 10:07:33.441942 (+ 11847us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:33.441953 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:33.441988 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:07:33.442112 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:33.442120 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:33.442509 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:33.442751 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:33.443913 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.445895 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.450295 (+ 4400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.451697 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.451834 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.452019 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:33.453160 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0924 10:07:33.453533 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11665,"thread_start_us":11557,"threads_started":1}
I20260924 10:07:33.457100 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.432948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49466 (local address 127.6.2.195:42961)
0924 10:07:33.449439 (+ 16491us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:33.449448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:33.449477 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:07:33.449564 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:33.449572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:33.449766 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:33.449979 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:33.450715 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.452340 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.454692 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.456031 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.456188 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.456414 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:33.456634 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 10:07:33.456780 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16330,"thread_start_us":16146,"threads_started":1}
I20260924 10:07:33.457095 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.432644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:42961 (local address 127.0.0.1:49466)
0924 10:07:33.448684 (+ 16040us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:33.448702 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:33.448824 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:33.449987 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:33.449994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:33.450018 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:33.450494 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.450506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.452505 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.452510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.454524 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.454536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.456229 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.456235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.456416 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.456454 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 10:07:33.456590 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 10:07:33.456711 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15899,"thread_start_us":16679,"threads_started":1}
I20260924 10:07:33.457700 8048 tablet_service.cc:1467] Processing CreateTablet for tablet f7dc31fc88c04f7bbf27f725173cc8a0 (DEFAULT_TABLE table=test-table [id=dcfd48f30beb4a32a1abde6984cfe321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:33.457829 8187 tablet_service.cc:1467] Processing CreateTablet for tablet f7dc31fc88c04f7bbf27f725173cc8a0 (DEFAULT_TABLE table=test-table [id=dcfd48f30beb4a32a1abde6984cfe321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:33.459180 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7dc31fc88c04f7bbf27f725173cc8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:33.460294 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7dc31fc88c04f7bbf27f725173cc8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:33.460033 8326 tablet_service.cc:1467] Processing CreateTablet for tablet f7dc31fc88c04f7bbf27f725173cc8a0 (DEFAULT_TABLE table=test-table [id=dcfd48f30beb4a32a1abde6984cfe321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:33.461136 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7dc31fc88c04f7bbf27f725173cc8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:33.471273 8415 tablet_bootstrap.cc:492] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70: Bootstrap starting.
I20260924 10:07:33.473500 8416 tablet_bootstrap.cc:492] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872: Bootstrap starting.
I20260924 10:07:33.475014 8415 tablet_bootstrap.cc:654] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:33.475279 8417 tablet_bootstrap.cc:492] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace: Bootstrap starting.
I20260924 10:07:33.476410 8415 log.cc:824] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:33.476804 8416 tablet_bootstrap.cc:654] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:33.477903 8416 log.cc:824] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:33.478392 8417 tablet_bootstrap.cc:654] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:33.479574 8417 log.cc:824] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:33.479882 8415 tablet_bootstrap.cc:492] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70: No bootstrap required, opened a new log
I20260924 10:07:33.480225 8415 ts_tablet_manager.cc:1397] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 10:07:33.482169 8416 tablet_bootstrap.cc:492] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872: No bootstrap required, opened a new log
I20260924 10:07:33.482426 8416 ts_tablet_manager.cc:1397] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260924 10:07:33.482430 8417 tablet_bootstrap.cc:492] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace: No bootstrap required, opened a new log
I20260924 10:07:33.482676 8417 ts_tablet_manager.cc:1397] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 10:07:33.489287 8416 raft_consensus.cc:348] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.489704 8416 raft_consensus.cc:374] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:33.489840 8416 raft_consensus.cc:729] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151858e25ef04c8c8e0f37c93e1b7872, State: Initialized, Role: FOLLOWER
I20260924 10:07:33.490513 8416 consensus_queue.cc:260] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.490607 8415 raft_consensus.cc:348] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.491000 8415 raft_consensus.cc:374] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:33.491118 8415 raft_consensus.cc:729] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307d18d9f9bb4d19baa09b4db2090b70, State: Initialized, Role: FOLLOWER
I20260924 10:07:33.491854 8257 heartbeater.cc:503] Master 127.6.2.254:35989 was elected leader, sending a full tablet report...
I20260924 10:07:33.491776 8415 consensus_queue.cc:260] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.493398 8417 raft_consensus.cc:348] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.493774 8417 raft_consensus.cc:374] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:33.493782 8396 heartbeater.cc:503] Master 127.6.2.254:35989 was elected leader, sending a full tablet report...
I20260924 10:07:33.493844 8416 ts_tablet_manager.cc:1428] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260924 10:07:33.493880 8417 raft_consensus.cc:729] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ce72fd9b574bdb9a37dd06cb75bace, State: Initialized, Role: FOLLOWER
I20260924 10:07:33.494489 8417 consensus_queue.cc:260] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.495903 8415 ts_tablet_manager.cc:1428] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260924 10:07:33.497375 8417 ts_tablet_manager.cc:1428] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260924 10:07:33.519757 8421 raft_consensus.cc:482] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:07:33.520095 8421 raft_consensus.cc:504] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.521937 8421 leader_election.cc:288] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193:39011), 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195:42961)
I20260924 10:07:33.529619 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.522354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:36939 (local address 127.6.2.195:42961)
0924 10:07:33.522600 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:33.522610 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:33.523520 (+ 910us) server_negotiation.cc:407] Connection header received
0924 10:07:33.523706 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:33.523718 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:33.523776 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:33.523887 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:33.524538 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.525586 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.527446 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.528641 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.528776 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.528904 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:33.529229 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 10:07:33.529374 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:07:33.529672 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.522484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:50597 (local address 127.6.2.193:39011)
0924 10:07:33.522700 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:33.522709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:33.523470 (+ 761us) server_negotiation.cc:407] Connection header received
0924 10:07:33.523685 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:33.523695 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:33.523763 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:33.523868 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:33.524468 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.525586 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.527639 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.528813 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.528950 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.529028 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:33.529344 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 10:07:33.529466 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:07:33.529764 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.522261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:39011 (local address 127.6.2.194:50597)
0924 10:07:33.523364 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:33.523379 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:33.523549 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:33.523881 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:33.523886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:33.523907 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:33.524327 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.524339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.525784 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.525787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.527489 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.527503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.529025 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.529031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.529187 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.529215 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:07:33.529298 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:07:33.529395 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":2,"thread_start_us":740,"threads_started":1}
I20260924 10:07:33.530009 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.522402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:42961 (local address 127.6.2.194:36939)
0924 10:07:33.523416 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:33.523430 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:33.523576 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:33.523898 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:33.523902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:33.523923 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:33.524382 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.524390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.525709 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.525712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.527276 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.527305 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.528836 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.528841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.529059 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.529095 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:07:33.529191 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 10:07:33.529325 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":27,"thread_start_us":518,"threads_started":1}
I20260924 10:07:33.530786 8073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7dc31fc88c04f7bbf27f725173cc8a0" candidate_uuid: "151858e25ef04c8c8e0f37c93e1b7872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" is_pre_election: true
I20260924 10:07:33.530912 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7dc31fc88c04f7bbf27f725173cc8a0" candidate_uuid: "151858e25ef04c8c8e0f37c93e1b7872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307d18d9f9bb4d19baa09b4db2090b70" is_pre_election: true
I20260924 10:07:33.531342 8073 raft_consensus.cc:2454] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151858e25ef04c8c8e0f37c93e1b7872 in term 0.
I20260924 10:07:33.531435 8351 raft_consensus.cc:2454] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151858e25ef04c8c8e0f37c93e1b7872 in term 0.
I20260924 10:07:33.532045 8141 leader_election.cc:302] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 151858e25ef04c8c8e0f37c93e1b7872, 307d18d9f9bb4d19baa09b4db2090b70, 97ce72fd9b574bdb9a37dd06cb75bace; no voters:
I20260924 10:07:33.532490 8421 raft_consensus.cc:2781] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:07:33.532675 8421 raft_consensus.cc:482] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:07:33.532778 8421 raft_consensus.cc:3037] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:33.534879 8421 raft_consensus.cc:504] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.535478 8421 leader_election.cc:288] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [CANDIDATE]: Term 1 election: Requested vote from peers 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193:39011), 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195:42961)
I20260924 10:07:33.535943 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7dc31fc88c04f7bbf27f725173cc8a0" candidate_uuid: "151858e25ef04c8c8e0f37c93e1b7872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307d18d9f9bb4d19baa09b4db2090b70"
I20260924 10:07:33.535985 8073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7dc31fc88c04f7bbf27f725173cc8a0" candidate_uuid: "151858e25ef04c8c8e0f37c93e1b7872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ce72fd9b574bdb9a37dd06cb75bace"
I20260924 10:07:33.536140 8351 raft_consensus.cc:3037] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:33.536183 8073 raft_consensus.cc:3037] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:33.538291 8351 raft_consensus.cc:2454] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151858e25ef04c8c8e0f37c93e1b7872 in term 1.
I20260924 10:07:33.538295 8073 raft_consensus.cc:2454] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151858e25ef04c8c8e0f37c93e1b7872 in term 1.
I20260924 10:07:33.538692 8139 leader_election.cc:302] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151858e25ef04c8c8e0f37c93e1b7872, 307d18d9f9bb4d19baa09b4db2090b70; no voters:
I20260924 10:07:33.539002 8421 raft_consensus.cc:2781] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:33.539435 8421 raft_consensus.cc:686] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 LEADER]: Becoming Leader. State: Replica: 151858e25ef04c8c8e0f37c93e1b7872, State: Running, Role: LEADER
I20260924 10:07:33.539956 8421 consensus_queue.cc:237] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.544462 7911 catalog_manager.cc:5495] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 reported cstate change: term changed from 0 to 1, leader changed from <none> to 151858e25ef04c8c8e0f37c93e1b7872 (127.6.2.194). New cstate: current_term: 1 leader_uuid: "151858e25ef04c8c8e0f37c93e1b7872" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } health_report { overall_health: UNKNOWN } } }
W20260924 10:07:33.584378 8119 tablet.cc:2369] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:07:33 dist-test-slave-s3n3 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244451, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:33.595674 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.581524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:35989 (local address 127.0.0.1:45182)
0924 10:07:33.582142 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:33.582159 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:33.582344 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:33.582685 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:33.582689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:33.583523 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:33.583820 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.583831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.584987 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.584996 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.586062 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.586075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.586881 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.586889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.587105 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.587765 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:33.587789 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:33.590193 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:33.593338 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:33.593348 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:33.593354 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:33.593794 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:33.594173 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:33.594177 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:33.594179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:33.594329 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:33.594822 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:33.594828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:33.594973 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 10:07:33.595125 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 10:07:33.595348 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":354,"threads_started":1}
I20260924 10:07:33.595811 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.581529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.6.2.254:35989)
0924 10:07:33.581715 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:33.581720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:33.582265 (+ 545us) server_negotiation.cc:407] Connection header received
0924 10:07:33.582485 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:33.582495 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:33.582551 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:33.582654 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:33.583977 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.584855 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.586212 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.586686 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.586840 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.590393 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:33.590399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:33.590402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:33.590455 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:33.593189 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:33.593903 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:33.593907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:33.593909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:33.594048 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:33.594435 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:33.594438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:33.594440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:33.594691 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:33.594826 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:33.595227 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 10:07:33.595484 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":62}
I20260924 10:07:33.600745 7911 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:45182:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:07:33.601878 7911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 10:07:33.602555 8258 tablet.cc:2369] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:07:33.607273 8397 tablet.cc:2369] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:07:33.609177 8187 tablet_service.cc:1467] Processing CreateTablet for tablet b94bc11a10d441479a66eb869d07e6fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5213352851ac485293852d3bdd008f56]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:07:33.608990 8048 tablet_service.cc:1467] Processing CreateTablet for tablet b94bc11a10d441479a66eb869d07e6fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5213352851ac485293852d3bdd008f56]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:07:33.609504 8326 tablet_service.cc:1467] Processing CreateTablet for tablet b94bc11a10d441479a66eb869d07e6fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5213352851ac485293852d3bdd008f56]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:07:33.610009 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94bc11a10d441479a66eb869d07e6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:33.610024 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94bc11a10d441479a66eb869d07e6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:33.610260 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94bc11a10d441479a66eb869d07e6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:33.615872 8416 tablet_bootstrap.cc:492] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872: Bootstrap starting.
I20260924 10:07:33.617163 8415 tablet_bootstrap.cc:492] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70: Bootstrap starting.
I20260924 10:07:33.617368 8416 tablet_bootstrap.cc:654] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:33.618726 8417 tablet_bootstrap.cc:492] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace: Bootstrap starting.
I20260924 10:07:33.618901 8415 tablet_bootstrap.cc:654] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:33.620101 8416 tablet_bootstrap.cc:492] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872: No bootstrap required, opened a new log
I20260924 10:07:33.620271 8416 ts_tablet_manager.cc:1397] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 10:07:33.620577 8417 tablet_bootstrap.cc:654] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:33.620976 8416 raft_consensus.cc:348] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.621223 8416 raft_consensus.cc:374] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:33.621387 8416 raft_consensus.cc:729] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151858e25ef04c8c8e0f37c93e1b7872, State: Initialized, Role: FOLLOWER
I20260924 10:07:33.621647 8416 consensus_queue.cc:260] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.622267 8416 ts_tablet_manager.cc:1428] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:07:33.622179 8421 tablet_replica.cc:439] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } } }
I20260924 10:07:33.622571 8421 tablet_replica.cc:442] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:33.626242 8417 tablet_bootstrap.cc:492] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace: No bootstrap required, opened a new log
I20260924 10:07:33.626420 8417 ts_tablet_manager.cc:1397] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 10:07:33.626623 8415 tablet_bootstrap.cc:492] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70: No bootstrap required, opened a new log
I20260924 10:07:33.626780 8415 ts_tablet_manager.cc:1397] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260924 10:07:33.627208 8417 raft_consensus.cc:348] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.627388 8417 raft_consensus.cc:374] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:33.627490 8417 raft_consensus.cc:729] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ce72fd9b574bdb9a37dd06cb75bace, State: Initialized, Role: FOLLOWER
I20260924 10:07:33.627503 8415 raft_consensus.cc:348] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.627763 8415 raft_consensus.cc:374] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:33.627724 8417 consensus_queue.cc:260] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.627861 8415 raft_consensus.cc:729] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307d18d9f9bb4d19baa09b4db2090b70, State: Initialized, Role: FOLLOWER
I20260924 10:07:33.628103 8415 consensus_queue.cc:260] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.628422 8417 ts_tablet_manager.cc:1428] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 10:07:33.628265 8423 tablet_replica.cc:439] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } } }
I20260924 10:07:33.628646 8415 ts_tablet_manager.cc:1428] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:07:33.628666 8423 tablet_replica.cc:442] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:33.628774 8422 tablet_replica.cc:439] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } } }
I20260924 10:07:33.629123 8422 tablet_replica.cc:442] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:33.888753 8421 raft_consensus.cc:482] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:07:33.888975 8421 raft_consensus.cc:504] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.889605 8421 leader_election.cc:288] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193:39011), 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195:42961)
I20260924 10:07:33.890249 8073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94bc11a10d441479a66eb869d07e6fc" candidate_uuid: "151858e25ef04c8c8e0f37c93e1b7872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" is_pre_election: true
I20260924 10:07:33.890290 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94bc11a10d441479a66eb869d07e6fc" candidate_uuid: "151858e25ef04c8c8e0f37c93e1b7872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307d18d9f9bb4d19baa09b4db2090b70" is_pre_election: true
I20260924 10:07:33.890564 8073 raft_consensus.cc:2454] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151858e25ef04c8c8e0f37c93e1b7872 in term 0.
I20260924 10:07:33.890580 8351 raft_consensus.cc:2454] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151858e25ef04c8c8e0f37c93e1b7872 in term 0.
I20260924 10:07:33.890985 8141 leader_election.cc:302] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151858e25ef04c8c8e0f37c93e1b7872, 97ce72fd9b574bdb9a37dd06cb75bace; no voters:
I20260924 10:07:33.891371 8421 raft_consensus.cc:2781] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:07:33.891511 8421 raft_consensus.cc:482] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:07:33.891651 8421 raft_consensus.cc:3037] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:33.894621 8421 raft_consensus.cc:504] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.895296 8421 leader_election.cc:288] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [CANDIDATE]: Term 1 election: Requested vote from peers 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193:39011), 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195:42961)
I20260924 10:07:33.895762 8073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94bc11a10d441479a66eb869d07e6fc" candidate_uuid: "151858e25ef04c8c8e0f37c93e1b7872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ce72fd9b574bdb9a37dd06cb75bace"
I20260924 10:07:33.895794 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94bc11a10d441479a66eb869d07e6fc" candidate_uuid: "151858e25ef04c8c8e0f37c93e1b7872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307d18d9f9bb4d19baa09b4db2090b70"
I20260924 10:07:33.895990 8073 raft_consensus.cc:3037] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:33.896008 8351 raft_consensus.cc:3037] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:33.897893 8351 raft_consensus.cc:2454] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151858e25ef04c8c8e0f37c93e1b7872 in term 1.
I20260924 10:07:33.897981 8073 raft_consensus.cc:2454] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151858e25ef04c8c8e0f37c93e1b7872 in term 1.
I20260924 10:07:33.898298 8139 leader_election.cc:302] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151858e25ef04c8c8e0f37c93e1b7872, 307d18d9f9bb4d19baa09b4db2090b70; no voters:
I20260924 10:07:33.898557 8421 raft_consensus.cc:2781] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:33.898768 8421 raft_consensus.cc:686] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 LEADER]: Becoming Leader. State: Replica: 151858e25ef04c8c8e0f37c93e1b7872, State: Running, Role: LEADER
I20260924 10:07:33.899016 8421 consensus_queue.cc:237] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:33.900566 8426 tablet_replica.cc:439] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872: TxnStatusTablet state changed. Reason: New leader 151858e25ef04c8c8e0f37c93e1b7872. Latest consensus state: current_term: 1 leader_uuid: "151858e25ef04c8c8e0f37c93e1b7872" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } } }
I20260924 10:07:33.900879 8426 tablet_replica.cc:442] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:07:33.902467 7909 catalog_manager.cc:5495] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 reported cstate change: term changed from 0 to 1, leader changed from <none> to 151858e25ef04c8c8e0f37c93e1b7872 (127.6.2.194). New cstate: current_term: 1 leader_uuid: "151858e25ef04c8c8e0f37c93e1b7872" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } health_report { overall_health: UNKNOWN } } }
I20260924 10:07:33.903537 8438 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:07:33.903836 8438 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:07:33.991950 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.984953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.6.2.194:34683)
0924 10:07:33.985808 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:33.985813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:33.985827 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:07:33.985899 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:33.985903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:33.985978 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:33.986106 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:33.986575 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.987433 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.989337 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.989863 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.990028 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.990536 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:33.991326 (+ 790us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:33.991450 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:33.991616 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 10:07:33.991691 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":615,"threads_started":1}
I20260924 10:07:33.992023 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:33.984807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:34683 (local address 127.0.0.1:45878)
0924 10:07:33.985040 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:33.985064 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:33.985232 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:33.986165 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:33.986170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:33.986184 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:07:33.986392 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.986403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.987558 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.987564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.989156 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:33.989171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:33.990037 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:33.990045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:33.990233 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:33.990305 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:07:33.991447 (+ 1142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:07:33.991468 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:07:33.991563 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:07:33.991729 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 10:07:33.999320 8351 raft_consensus.cc:1261] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Refusing update from remote peer 151858e25ef04c8c8e0f37c93e1b7872: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:33.999383 8073 raft_consensus.cc:1261] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Refusing update from remote peer 151858e25ef04c8c8e0f37c93e1b7872: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:34.000293 8426 consensus_queue.cc:1035] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:34.000702 8421 consensus_queue.cc:1035] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:34.009433 8422 tablet_replica.cc:439] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70: TxnStatusTablet state changed. Reason: New leader 151858e25ef04c8c8e0f37c93e1b7872. Latest consensus state: current_term: 1 leader_uuid: "151858e25ef04c8c8e0f37c93e1b7872" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } } }
I20260924 10:07:34.009693 8422 tablet_replica.cc:442] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:34.011994 8442 mvcc.cc:205] Tried to move back new op lower bound from 7332841283571638272 to 7332841283172868096. Current Snapshot: MvccSnapshot[applied={T|T < 7332841283571638272}]
I20260924 10:07:34.017505 8421 consensus_queue.cc:1035] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:34.017532 8423 tablet_replica.cc:439] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace: TxnStatusTablet state changed. Reason: New leader 151858e25ef04c8c8e0f37c93e1b7872. Latest consensus state: current_term: 1 leader_uuid: "151858e25ef04c8c8e0f37c93e1b7872" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } } }
I20260924 10:07:34.017819 8423 tablet_replica.cc:442] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:34.020258 8422 tablet_replica.cc:439] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "151858e25ef04c8c8e0f37c93e1b7872" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } } }
I20260924 10:07:34.020530 8422 tablet_replica.cc:442] T b94bc11a10d441479a66eb869d07e6fc P 307d18d9f9bb4d19baa09b4db2090b70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:34.023895 8421 tablet_replica.cc:439] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "151858e25ef04c8c8e0f37c93e1b7872" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } } }
I20260924 10:07:34.024475 8421 tablet_replica.cc:442] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:07:34.027738 8421 tablet_replica.cc:439] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "151858e25ef04c8c8e0f37c93e1b7872" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } } }
I20260924 10:07:34.028038 8421 tablet_replica.cc:442] T b94bc11a10d441479a66eb869d07e6fc P 151858e25ef04c8c8e0f37c93e1b7872: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:07:34.031323 8423 tablet_replica.cc:439] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "151858e25ef04c8c8e0f37c93e1b7872" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } } }
I20260924 10:07:34.031564 8423 tablet_replica.cc:442] T b94bc11a10d441479a66eb869d07e6fc P 97ce72fd9b574bdb9a37dd06cb75bace: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:07:34.042671 8421 consensus_queue.cc:1035] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:34.047935 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:34.038907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.6.2.194:34683)
0924 10:07:34.039121 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:34.039131 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:34.039711 (+ 580us) server_negotiation.cc:407] Connection header received
0924 10:07:34.040021 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:34.040027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:34.040110 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:34.040215 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:34.043223 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.044018 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.045417 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.046074 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.046217 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:34.046810 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:34.046969 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:34.047505 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:34.047570 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 10:07:34.047655 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 10:07:34.065965 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:34.059473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:36169 (local address 127.6.2.194:34683)
0924 10:07:34.059663 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:34.059672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:34.060307 (+ 635us) server_negotiation.cc:407] Connection header received
0924 10:07:34.060503 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:34.060506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:34.060588 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:34.060665 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:34.061256 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.061992 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.064101 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.064579 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.064674 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:34.065197 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:34.065302 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:34.065371 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:34.065622 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 10:07:34.065706 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":61}
I20260924 10:07:34.066217 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:34.059502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:34683 (local address 127.6.2.194:36169)
0924 10:07:34.060201 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:34.060221 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:34.060366 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:34.060708 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:34.060713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:34.060728 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:07:34.061083 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:34.061094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.062124 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.062129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:34.063956 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:34.063970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.064726 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.064733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:34.064923 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:34.064993 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:07:34.065424 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:07:34.065435 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 10:07:34.065550 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 10:07:34.065722 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":27,"thread_start_us":337,"threads_started":1}
I20260924 10:07:34.137760 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:34.138063 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:34.141639 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc: Sending DeleteTablet for 3 replicas of tablet f7dc31fc88c04f7bbf27f725173cc8a0
I20260924 10:07:34.142838 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet f7dc31fc88c04f7bbf27f725173cc8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:34 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:45864
I20260924 10:07:34.142874 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet f7dc31fc88c04f7bbf27f725173cc8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:34 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:46564
I20260924 10:07:34.142933 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet f7dc31fc88c04f7bbf27f725173cc8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:34 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:49466
I20260924 10:07:34.146636 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:07:34.147531 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:07:34.155786 8463 tablet_replica.cc:330] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70: stopping tablet replica
I20260924 10:07:34.156025 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2c5505479b4165a06ac17057062178 (DEFAULT_TABLE table=test-table [id=05736e6977c64cf7976e9f90d827c599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:34.156257 8463 raft_consensus.cc:2229] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:34.156479 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2c5505479b4165a06ac17057062178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:34.156466 8187 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2c5505479b4165a06ac17057062178 (DEFAULT_TABLE table=test-table [id=05736e6977c64cf7976e9f90d827c599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:34.156733 8463 raft_consensus.cc:2258] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:34.156883 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2c5505479b4165a06ac17057062178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:34.159085 8325 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2c5505479b4165a06ac17057062178 (DEFAULT_TABLE table=test-table [id=05736e6977c64cf7976e9f90d827c599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:34.159153 8463 ts_tablet_manager.cc:1905] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:34.159574 8325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2c5505479b4165a06ac17057062178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:34.161686 8462 tablet_replica.cc:330] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace: stopping tablet replica
I20260924 10:07:34.162101 8462 raft_consensus.cc:2229] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:34.162439 8462 raft_consensus.cc:2258] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:34.169090 8462 ts_tablet_manager.cc:1905] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:34.172690 8461 tablet_replica.cc:330] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872: stopping tablet replica
I20260924 10:07:34.173909 8461 raft_consensus.cc:2229] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:07:34.175146 8461 raft_consensus.cc:2258] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:34.175488 8465 tablet_bootstrap.cc:492] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace: Bootstrap starting.
I20260924 10:07:34.177191 8461 ts_tablet_manager.cc:1905] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:34.177448 8465 tablet_bootstrap.cc:654] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:34.177438 8464 tablet_bootstrap.cc:492] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872: Bootstrap starting.
I20260924 10:07:34.180900 8464 tablet_bootstrap.cc:654] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:34.182888 8463 ts_tablet_manager.cc:1918] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:07:34.183161 8463 log.cc:1196] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/f7dc31fc88c04f7bbf27f725173cc8a0
I20260924 10:07:34.183813 8463 ts_tablet_manager.cc:1939] T f7dc31fc88c04f7bbf27f725173cc8a0 P 307d18d9f9bb4d19baa09b4db2090b70: Deleting consensus metadata
I20260924 10:07:34.185896 7896 catalog_manager.cc:4841] TS 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195:42961): tablet f7dc31fc88c04f7bbf27f725173cc8a0 (table test-table [id=dcfd48f30beb4a32a1abde6984cfe321]) successfully deleted
I20260924 10:07:34.187569 8466 tablet_bootstrap.cc:492] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70: Bootstrap starting.
I20260924 10:07:34.188982 8465 tablet_bootstrap.cc:492] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace: No bootstrap required, opened a new log
I20260924 10:07:34.189141 8465 ts_tablet_manager.cc:1397] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260924 10:07:34.189934 8465 raft_consensus.cc:348] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:34.190131 8462 ts_tablet_manager.cc:1918] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:07:34.190197 8465 raft_consensus.cc:374] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:34.190323 8465 raft_consensus.cc:729] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ce72fd9b574bdb9a37dd06cb75bace, State: Initialized, Role: FOLLOWER
I20260924 10:07:34.190358 8462 log.cc:1196] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/f7dc31fc88c04f7bbf27f725173cc8a0
I20260924 10:07:34.190629 8465 consensus_queue.cc:260] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:34.191150 8462 ts_tablet_manager.cc:1939] T f7dc31fc88c04f7bbf27f725173cc8a0 P 97ce72fd9b574bdb9a37dd06cb75bace: Deleting consensus metadata
I20260924 10:07:34.191299 8465 ts_tablet_manager.cc:1428] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 10:07:34.192369 8464 tablet_bootstrap.cc:492] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872: No bootstrap required, opened a new log
I20260924 10:07:34.192514 8464 ts_tablet_manager.cc:1397] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20260924 10:07:34.192675 8466 tablet_bootstrap.cc:654] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:34.193349 8464 raft_consensus.cc:348] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:34.193536 7898 catalog_manager.cc:4841] TS 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193:39011): tablet f7dc31fc88c04f7bbf27f725173cc8a0 (table test-table [id=dcfd48f30beb4a32a1abde6984cfe321]) successfully deleted
I20260924 10:07:34.193637 8464 raft_consensus.cc:374] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:34.193763 8464 raft_consensus.cc:729] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151858e25ef04c8c8e0f37c93e1b7872, State: Initialized, Role: FOLLOWER
I20260924 10:07:34.194054 8464 consensus_queue.cc:260] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:34.194818 8464 ts_tablet_manager.cc:1428] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 10:07:34.195024 8461 ts_tablet_manager.cc:1918] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:07:34.195243 8461 log.cc:1196] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/f7dc31fc88c04f7bbf27f725173cc8a0
I20260924 10:07:34.195914 8461 ts_tablet_manager.cc:1939] T f7dc31fc88c04f7bbf27f725173cc8a0 P 151858e25ef04c8c8e0f37c93e1b7872: Deleting consensus metadata
I20260924 10:07:34.198217 7896 catalog_manager.cc:4841] TS 151858e25ef04c8c8e0f37c93e1b7872 (127.6.2.194:34683): tablet f7dc31fc88c04f7bbf27f725173cc8a0 (table test-table [id=dcfd48f30beb4a32a1abde6984cfe321]) successfully deleted
I20260924 10:07:34.200752 8466 tablet_bootstrap.cc:492] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70: No bootstrap required, opened a new log
I20260924 10:07:34.200927 8466 ts_tablet_manager.cc:1397] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20260924 10:07:34.201545 8466 raft_consensus.cc:348] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:34.201716 8466 raft_consensus.cc:374] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:34.201830 8466 raft_consensus.cc:729] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307d18d9f9bb4d19baa09b4db2090b70, State: Initialized, Role: FOLLOWER
I20260924 10:07:34.202080 8466 consensus_queue.cc:260] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:34.202617 8466 ts_tablet_manager.cc:1428] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:07:34.305742 8449 raft_consensus.cc:482] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:07:34.306064 8449 raft_consensus.cc:504] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:34.307855 8449 leader_election.cc:288] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193:39011), 151858e25ef04c8c8e0f37c93e1b7872 (127.6.2.194:34683)
I20260924 10:07:34.328364 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:34.308150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:39011 (local address 127.6.2.195:41953)
0924 10:07:34.320729 (+ 12579us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:34.320759 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:34.320958 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:34.321588 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:34.321593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:34.321624 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:34.322098 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:34.322111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.323747 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.323752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:34.325869 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:34.325884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.327604 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.327610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:34.327736 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:34.327759 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:07:34.327870 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 10:07:34.327999 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12383,"thread_start_us":13241,"threads_started":1}
I20260924 10:07:34.328395 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:34.308428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:41953 (local address 127.6.2.193:39011)
0924 10:07:34.321005 (+ 12577us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:34.321012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:34.321040 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:07:34.321116 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:34.321122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:34.321200 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:34.321336 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:34.322284 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.323587 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.326020 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.327251 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.327367 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:34.327502 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:34.327894 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 10:07:34.328024 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12442,"thread_start_us":12643,"threads_started":1}
I20260924 10:07:34.329470 8073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2c5505479b4165a06ac17057062178" candidate_uuid: "307d18d9f9bb4d19baa09b4db2090b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" is_pre_election: true
I20260924 10:07:34.329754 8073 raft_consensus.cc:2454] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 307d18d9f9bb4d19baa09b4db2090b70 in term 0.
I20260924 10:07:34.329833 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:34.308833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:46723 (local address 127.6.2.194:34683)
0924 10:07:34.309047 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:34.309054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:34.321909 (+ 12855us) server_negotiation.cc:407] Connection header received
0924 10:07:34.322100 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:34.322107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:34.322180 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:34.322290 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:34.322832 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.323867 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.325782 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.327154 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.327310 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:34.327435 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:34.329440 (+ 2005us) server_negotiation.cc:299] Negotiation successful
0924 10:07:34.329580 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:07:34.330453 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:34.308345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:34683 (local address 127.6.2.195:46723)
0924 10:07:34.321810 (+ 13465us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:34.321830 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:34.321960 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:34.322308 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:34.322313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:34.322338 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:34.322660 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:34.322671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.324006 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.324015 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:34.325644 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:34.325655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.329063 (+ 3408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.329078 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:34.329245 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:34.329278 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:07:34.329985 (+ 707us) client_negotiation.cc:240] Negotiation successful
0924 10:07:34.330112 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13190,"thread_start_us":11975,"threads_started":1}
I20260924 10:07:34.330943 8280 leader_election.cc:302] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307d18d9f9bb4d19baa09b4db2090b70, 97ce72fd9b574bdb9a37dd06cb75bace; no voters:
I20260924 10:07:34.331441 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2c5505479b4165a06ac17057062178" candidate_uuid: "307d18d9f9bb4d19baa09b4db2090b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "151858e25ef04c8c8e0f37c93e1b7872" is_pre_election: true
I20260924 10:07:34.331851 8207 raft_consensus.cc:2454] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 307d18d9f9bb4d19baa09b4db2090b70 in term 0.
I20260924 10:07:34.332022 8449 raft_consensus.cc:2781] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:07:34.332126 8449 raft_consensus.cc:482] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:07:34.332206 8449 raft_consensus.cc:3037] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:34.334740 8449 raft_consensus.cc:504] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:34.335465 8449 leader_election.cc:288] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [CANDIDATE]: Term 1 election: Requested vote from peers 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193:39011), 151858e25ef04c8c8e0f37c93e1b7872 (127.6.2.194:34683)
I20260924 10:07:34.335948 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2c5505479b4165a06ac17057062178" candidate_uuid: "307d18d9f9bb4d19baa09b4db2090b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "151858e25ef04c8c8e0f37c93e1b7872"
I20260924 10:07:34.336150 8207 raft_consensus.cc:3037] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:34.336407 8073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2c5505479b4165a06ac17057062178" candidate_uuid: "307d18d9f9bb4d19baa09b4db2090b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ce72fd9b574bdb9a37dd06cb75bace"
I20260924 10:07:34.336627 8073 raft_consensus.cc:3037] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:34.339522 8207 raft_consensus.cc:2454] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 307d18d9f9bb4d19baa09b4db2090b70 in term 1.
I20260924 10:07:34.339723 8073 raft_consensus.cc:2454] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 307d18d9f9bb4d19baa09b4db2090b70 in term 1.
I20260924 10:07:34.340122 8280 leader_election.cc:302] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307d18d9f9bb4d19baa09b4db2090b70, 97ce72fd9b574bdb9a37dd06cb75bace; no voters:
I20260924 10:07:34.340456 8449 raft_consensus.cc:2781] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:34.340629 8449 raft_consensus.cc:686] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 LEADER]: Becoming Leader. State: Replica: 307d18d9f9bb4d19baa09b4db2090b70, State: Running, Role: LEADER
I20260924 10:07:34.341055 8449 consensus_queue.cc:237] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:34.344555 7909 catalog_manager.cc:5495] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195). New cstate: current_term: 1 leader_uuid: "307d18d9f9bb4d19baa09b4db2090b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } health_report { overall_health: HEALTHY } } }
I20260924 10:07:34.393105 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:34.386147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.6.2.195:42961)
0924 10:07:34.386690 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:34.386694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:34.386712 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:07:34.386786 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:34.386790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:34.386864 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:34.386998 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:34.387432 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.388192 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.390209 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.390853 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.390985 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:34.391665 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:34.392455 (+ 790us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:34.392601 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:34.392662 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 10:07:34.392744 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":316,"threads_started":1}
I20260924 10:07:34.409602 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:34.403102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:34683 (local address 127.6.2.195:42309)
0924 10:07:34.403816 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:34.403832 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:34.403943 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:34.404331 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:34.404334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:34.404344 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:07:34.404582 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:34.404590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.406193 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.406196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:34.407428 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:34.407439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.408528 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.408532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:34.408652 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:34.408703 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:07:34.409161 (+ 458us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:07:34.409169 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:07:34.409228 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 10:07:34.409332 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":372,"threads_started":1}
I20260924 10:07:34.409716 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:34.403156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:42309 (local address 127.6.2.194:34683)
0924 10:07:34.403357 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:34.403365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:34.403918 (+ 553us) server_negotiation.cc:407] Connection header received
0924 10:07:34.404068 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:34.404074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:34.404188 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:34.404298 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:34.404739 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.406030 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.407561 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.408344 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.408474 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:34.408879 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:34.409058 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:34.409159 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:34.409282 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 10:07:34.409381 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:07:34.418776 8073 raft_consensus.cc:1261] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Refusing update from remote peer 307d18d9f9bb4d19baa09b4db2090b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:34.419454 8449 consensus_queue.cc:1035] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:34.419661 8207 raft_consensus.cc:1261] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Refusing update from remote peer 307d18d9f9bb4d19baa09b4db2090b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:34.420308 8449 consensus_queue.cc:1035] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:34.424688 8443 mvcc.cc:205] Tried to move back new op lower bound from 7332841285293600768 to 7332841284984893440. Current Snapshot: MvccSnapshot[applied={T|T < 7332841285293600768}]
I20260924 10:07:34.429100 8444 mvcc.cc:205] Tried to move back new op lower bound from 7332841285293600768 to 7332841284984893440. Current Snapshot: MvccSnapshot[applied={T|T < 7332841285293600768}]
I20260924 10:07:34.453194 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:34.447202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:48067 (local address 127.6.2.195:42961)
0924 10:07:34.447370 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:34.447376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:34.447393 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:07:34.447448 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:34.447453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:34.447529 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:34.447626 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:34.448080 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.449097 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.451289 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.451924 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.452118 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:34.452554 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:34.452645 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:34.452711 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:34.452873 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 10:07:34.452930 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:07:34.453395 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:34.446868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:42961 (local address 127.6.2.194:48067)
0924 10:07:34.447081 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:34.447104 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:34.447228 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:34.447671 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:34.447678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:34.447712 (+ 34us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:07:34.447944 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:34.447960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.449236 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.449242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:34.451133 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:34.451150 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:34.452152 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:34.452159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:34.452359 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:34.452412 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:07:34.452752 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:07:34.452764 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 10:07:34.452864 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 10:07:34.453029 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 10:07:34.454612 8443 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 10:07:36.051522 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:36.051746 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:36.055747 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc: Sending DeleteTablet for 3 replicas of tablet 9b2c5505479b4165a06ac17057062178
I20260924 10:07:36.056517 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2c5505479b4165a06ac17057062178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:36 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:46564
I20260924 10:07:36.056507 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2c5505479b4165a06ac17057062178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:36 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:45864
I20260924 10:07:36.056651 8325 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2c5505479b4165a06ac17057062178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:36 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:49466
I20260924 10:07:36.057292 8508 tablet_replica.cc:330] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872: stopping tablet replica
I20260924 10:07:36.057744 8508 raft_consensus.cc:2229] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:36.058142 8508 raft_consensus.cc:2258] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:36.058835 8509 tablet_replica.cc:330] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace: stopping tablet replica
I20260924 10:07:36.058990 8510 tablet_replica.cc:330] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70: stopping tablet replica
I20260924 10:07:36.059134 8509 raft_consensus.cc:2229] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:36.059334 8510 raft_consensus.cc:2229] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:07:36.059427 8509 raft_consensus.cc:2258] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:36.060005 8510 raft_consensus.cc:2258] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:36.061120 8510 ts_tablet_manager.cc:1905] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:36.064572 8508 ts_tablet_manager.cc:1905] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:36.067974 8509 ts_tablet_manager.cc:1905] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:36.068773 8510 ts_tablet_manager.cc:1918] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:07:36.068969 8510 log.cc:1196] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/9b2c5505479b4165a06ac17057062178
I20260924 10:07:36.069496 8510 ts_tablet_manager.cc:1939] T 9b2c5505479b4165a06ac17057062178 P 307d18d9f9bb4d19baa09b4db2090b70: Deleting consensus metadata
I20260924 10:07:36.070767 7896 catalog_manager.cc:4841] TS 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195:42961): tablet 9b2c5505479b4165a06ac17057062178 (table test-table [id=05736e6977c64cf7976e9f90d827c599]) successfully deleted
I20260924 10:07:36.071271 8508 ts_tablet_manager.cc:1918] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:07:36.071403 8508 log.cc:1196] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/9b2c5505479b4165a06ac17057062178
I20260924 10:07:36.071851 8508 ts_tablet_manager.cc:1939] T 9b2c5505479b4165a06ac17057062178 P 151858e25ef04c8c8e0f37c93e1b7872: Deleting consensus metadata
I20260924 10:07:36.073011 7896 catalog_manager.cc:4841] TS 151858e25ef04c8c8e0f37c93e1b7872 (127.6.2.194:34683): tablet 9b2c5505479b4165a06ac17057062178 (table test-table [id=05736e6977c64cf7976e9f90d827c599]) successfully deleted
I20260924 10:07:36.074107 8509 ts_tablet_manager.cc:1918] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:07:36.074259 8509 log.cc:1196] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/9b2c5505479b4165a06ac17057062178
I20260924 10:07:36.074623 8509 ts_tablet_manager.cc:1939] T 9b2c5505479b4165a06ac17057062178 P 97ce72fd9b574bdb9a37dd06cb75bace: Deleting consensus metadata
I20260924 10:07:36.075568 7898 catalog_manager.cc:4841] TS 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193:39011): tablet 9b2c5505479b4165a06ac17057062178 (table test-table [id=05736e6977c64cf7976e9f90d827c599]) successfully deleted
Sep 24 10:07:36 dist-test-slave-s3n3 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:07:36 dist-test-slave-s3n3 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:36.106958 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:36.093276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45198 (local address 127.6.2.254:35989)
0924 10:07:36.093815 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:36.093819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:36.093841 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:07:36.093894 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:36.093897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:36.093962 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:36.094057 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:36.095404 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.096190 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.097426 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.097814 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.097913 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:36.101973 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:36.101978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:36.101981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:36.102012 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:36.104556 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:36.105144 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:36.105148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:36.105150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:36.105264 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:36.105558 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:36.105561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:36.105563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:36.105803 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:36.106081 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:36.106382 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 10:07:36.106594 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":286,"threads_started":1}
I20260924 10:07:36.110144 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:07:36.110661 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:07:36.116781 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 17824395f88f46ea83961f5e677a71af (DEFAULT_TABLE table=test-table [id=9d9df07eab0348c49b4631a25e1ab146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:36.116886 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 17824395f88f46ea83961f5e677a71af (DEFAULT_TABLE table=test-table [id=9d9df07eab0348c49b4631a25e1ab146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:36.117100 8325 tablet_service.cc:1467] Processing CreateTablet for tablet 17824395f88f46ea83961f5e677a71af (DEFAULT_TABLE table=test-table [id=9d9df07eab0348c49b4631a25e1ab146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:36.117277 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17824395f88f46ea83961f5e677a71af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:36.117342 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17824395f88f46ea83961f5e677a71af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:36.117542 8325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17824395f88f46ea83961f5e677a71af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:36.123814 8519 tablet_bootstrap.cc:492] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace: Bootstrap starting.
I20260924 10:07:36.124483 8521 tablet_bootstrap.cc:492] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70: Bootstrap starting.
I20260924 10:07:36.125381 8520 tablet_bootstrap.cc:492] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872: Bootstrap starting.
I20260924 10:07:36.125672 8519 tablet_bootstrap.cc:654] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:36.126261 8521 tablet_bootstrap.cc:654] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:36.127051 8520 tablet_bootstrap.cc:654] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:36.128580 8519 tablet_bootstrap.cc:492] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace: No bootstrap required, opened a new log
I20260924 10:07:36.128712 8521 tablet_bootstrap.cc:492] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70: No bootstrap required, opened a new log
I20260924 10:07:36.128749 8519 ts_tablet_manager.cc:1397] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 10:07:36.128860 8521 ts_tablet_manager.cc:1397] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 10:07:36.129267 8520 tablet_bootstrap.cc:492] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872: No bootstrap required, opened a new log
I20260924 10:07:36.129412 8520 ts_tablet_manager.cc:1397] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 10:07:36.129587 8521 raft_consensus.cc:348] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.129563 8519 raft_consensus.cc:348] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.129801 8519 raft_consensus.cc:374] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:36.129802 8521 raft_consensus.cc:374] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:36.129911 8519 raft_consensus.cc:729] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ce72fd9b574bdb9a37dd06cb75bace, State: Initialized, Role: FOLLOWER
I20260924 10:07:36.129912 8521 raft_consensus.cc:729] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307d18d9f9bb4d19baa09b4db2090b70, State: Initialized, Role: FOLLOWER
I20260924 10:07:36.130177 8521 consensus_queue.cc:260] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.130254 8520 raft_consensus.cc:348] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.130322 8519 consensus_queue.cc:260] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.130522 8520 raft_consensus.cc:374] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:36.130609 8520 raft_consensus.cc:729] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151858e25ef04c8c8e0f37c93e1b7872, State: Initialized, Role: FOLLOWER
I20260924 10:07:36.130944 8520 consensus_queue.cc:260] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.131645 8519 ts_tablet_manager.cc:1428] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 10:07:36.132125 8520 ts_tablet_manager.cc:1428] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 10:07:36.133086 8521 ts_tablet_manager.cc:1428] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 10:07:36.141827 8525 raft_consensus.cc:482] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:07:36.142015 8525 raft_consensus.cc:504] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.143532 8525 leader_election.cc:288] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 151858e25ef04c8c8e0f37c93e1b7872 (127.6.2.194:34683), 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195:42961)
I20260924 10:07:36.152048 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:36.144519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:40481 (local address 127.6.2.195:42961)
0924 10:07:36.145663 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:36.145670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:36.145694 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:07:36.145753 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:36.145758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:36.145829 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:36.145938 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:36.146532 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.147312 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.149596 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.150807 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.150942 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:36.151051 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:36.151577 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 10:07:36.151701 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":899,"threads_started":1}
I20260924 10:07:36.152037 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:36.143811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:47493 (local address 127.6.2.194:34683)
0924 10:07:36.145174 (+ 1363us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:36.145180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:36.145203 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:07:36.145268 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:36.145272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:36.145341 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:36.145441 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:36.146108 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.147020 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.149576 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.150808 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.150942 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:36.151048 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:36.151563 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 10:07:36.151701 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":1091,"threads_started":1}
I20260924 10:07:36.152226 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:36.143772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:34683 (local address 127.6.2.193:47493)
0924 10:07:36.144430 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:36.144457 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:36.144614 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:36.145577 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:36.145581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:36.145609 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:36.145913 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:36.145928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.147195 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.147204 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:36.149386 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:36.149400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.151029 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.151034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:36.151327 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:36.151435 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 10:07:36.151530 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:07:36.151689 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":60,"thread_start_us":300,"threads_started":1}
I20260924 10:07:36.152565 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:36.144437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:42961 (local address 127.6.2.193:40481)
0924 10:07:36.145414 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:36.145432 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:36.145532 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:36.145953 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:36.145955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:36.145976 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:36.146379 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:36.146388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.147436 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.147441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:36.149458 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:36.149471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.151012 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.151019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:36.151331 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:36.151441 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 10:07:36.151530 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 10:07:36.151689 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":178,"thread_start_us":668,"threads_started":1}
I20260924 10:07:36.152871 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17824395f88f46ea83961f5e677a71af" candidate_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "151858e25ef04c8c8e0f37c93e1b7872" is_pre_election: true
I20260924 10:07:36.153128 8207 raft_consensus.cc:2454] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ce72fd9b574bdb9a37dd06cb75bace in term 0.
I20260924 10:07:36.153127 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17824395f88f46ea83961f5e677a71af" candidate_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307d18d9f9bb4d19baa09b4db2090b70" is_pre_election: true
I20260924 10:07:36.153393 8351 raft_consensus.cc:2454] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ce72fd9b574bdb9a37dd06cb75bace in term 0.
I20260924 10:07:36.153647 7999 leader_election.cc:302] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151858e25ef04c8c8e0f37c93e1b7872, 97ce72fd9b574bdb9a37dd06cb75bace; no voters:
I20260924 10:07:36.154017 8525 raft_consensus.cc:2781] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:07:36.154143 8525 raft_consensus.cc:482] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:07:36.154244 8525 raft_consensus.cc:3037] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:36.156558 8525 raft_consensus.cc:504] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.157193 8525 leader_election.cc:288] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [CANDIDATE]: Term 1 election: Requested vote from peers 151858e25ef04c8c8e0f37c93e1b7872 (127.6.2.194:34683), 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195:42961)
I20260924 10:07:36.157629 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17824395f88f46ea83961f5e677a71af" candidate_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "151858e25ef04c8c8e0f37c93e1b7872"
I20260924 10:07:36.157749 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17824395f88f46ea83961f5e677a71af" candidate_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307d18d9f9bb4d19baa09b4db2090b70"
I20260924 10:07:36.157837 8207 raft_consensus.cc:3037] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:36.157963 8351 raft_consensus.cc:3037] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:36.160812 8351 raft_consensus.cc:2454] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ce72fd9b574bdb9a37dd06cb75bace in term 1.
I20260924 10:07:36.160825 8207 raft_consensus.cc:2454] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ce72fd9b574bdb9a37dd06cb75bace in term 1.
I20260924 10:07:36.161171 7999 leader_election.cc:302] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307d18d9f9bb4d19baa09b4db2090b70, 97ce72fd9b574bdb9a37dd06cb75bace; no voters:
I20260924 10:07:36.161468 8525 raft_consensus.cc:2781] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:36.161895 8525 raft_consensus.cc:686] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 LEADER]: Becoming Leader. State: Replica: 97ce72fd9b574bdb9a37dd06cb75bace, State: Running, Role: LEADER
I20260924 10:07:36.162274 8525 consensus_queue.cc:237] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.165652 7909 catalog_manager.cc:5495] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace reported cstate change: term changed from 0 to 1, leader changed from <none> to 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193). New cstate: current_term: 1 leader_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } health_report { overall_health: UNKNOWN } } }
I20260924 10:07:36.198377 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:36.192727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46572 (local address 127.6.2.193:39011)
0924 10:07:36.193302 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:36.193306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:36.193335 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:07:36.193416 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:36.193420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:36.193489 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:36.193656 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:36.194082 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.194851 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.196165 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.196577 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.196705 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:36.197149 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:36.197745 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:36.197905 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:36.197971 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 10:07:36.198055 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":300,"threads_started":1}
I20260924 10:07:36.212001 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:36.207384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:39227 (local address 127.6.2.194:34683)
0924 10:07:36.207542 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:36.207547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:36.207980 (+ 433us) server_negotiation.cc:407] Connection header received
0924 10:07:36.208121 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:36.208125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:36.208185 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:36.208263 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:36.208668 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.209343 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.210561 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.210955 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.211042 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:36.211468 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:36.211555 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:36.211620 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:36.211735 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 10:07:36.211791 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:07:36.212046 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:36.207222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:34683 (local address 127.6.2.193:39227)
0924 10:07:36.207892 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:36.207912 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:36.208020 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:36.208289 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:36.208292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:36.208300 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:07:36.208555 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:36.208561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.209456 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.209459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:36.210450 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:36.210459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.211140 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.211147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:36.211277 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:36.211336 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:07:36.211635 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:07:36.211642 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:07:36.211699 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 10:07:36.211785 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":361,"threads_started":1}
I20260924 10:07:36.219664 8207 raft_consensus.cc:1261] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Refusing update from remote peer 97ce72fd9b574bdb9a37dd06cb75bace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:36.219707 8351 raft_consensus.cc:1261] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Refusing update from remote peer 97ce72fd9b574bdb9a37dd06cb75bace: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:36.220260 8530 consensus_queue.cc:1035] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [LEADER]: Connected to new peer: Peer: permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:36.220603 8525 consensus_queue.cc:1035] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [LEADER]: Connected to new peer: Peer: permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:36.251955 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:36.245283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:39011 (local address 127.6.2.194:47527)
0924 10:07:36.246072 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:36.246089 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:36.246215 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:36.246622 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:36.246627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:36.246642 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:07:36.247218 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:36.247237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.248351 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.248357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:36.249526 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:36.249538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.250433 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.250439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:36.250622 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:36.250674 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:07:36.251339 (+ 665us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:07:36.251349 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:07:36.251425 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 10:07:36.251586 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":485,"threads_started":1}
I20260924 10:07:36.251921 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:36.245321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:47527 (local address 127.6.2.193:39011)
0924 10:07:36.245690 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:36.245701 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:36.246190 (+ 489us) server_negotiation.cc:407] Connection header received
0924 10:07:36.246368 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:36.246376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:36.246478 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:36.246600 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:36.247425 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.248168 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.249675 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.250225 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.250381 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:36.250933 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:36.251202 (+ 269us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:36.251330 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:36.251465 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 10:07:36.251631 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 10:07:36.281358 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:36.281621 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:36.284584 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc: Sending DeleteTablet for 3 replicas of tablet 17824395f88f46ea83961f5e677a71af
I20260924 10:07:36.285323 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 17824395f88f46ea83961f5e677a71af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:36 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:46564
I20260924 10:07:36.285480 8325 tablet_service.cc:1514] Processing DeleteTablet for tablet 17824395f88f46ea83961f5e677a71af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:36 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:49466
I20260924 10:07:36.285625 8509 tablet_replica.cc:330] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace: stopping tablet replica
I20260924 10:07:36.285741 8510 tablet_replica.cc:330] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70: stopping tablet replica
I20260924 10:07:36.285738 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 17824395f88f46ea83961f5e677a71af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:36 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:45864
I20260924 10:07:36.285838 8509 raft_consensus.cc:2229] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:07:36.285940 8510 raft_consensus.cc:2229] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:36.286115 8510 raft_consensus.cc:2258] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:36.286178 8508 tablet_replica.cc:330] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872: stopping tablet replica
I20260924 10:07:36.286315 8509 raft_consensus.cc:2258] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:36.286370 8508 raft_consensus.cc:2229] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:36.286551 8508 raft_consensus.cc:2258] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:36.287325 8510 ts_tablet_manager.cc:1905] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:36.287679 8509 ts_tablet_manager.cc:1905] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:36.288004 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:07:36.288621 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:07:36.291283 8508 ts_tablet_manager.cc:1905] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:36.293843 8510 ts_tablet_manager.cc:1918] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:07:36.293994 8510 log.cc:1196] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/17824395f88f46ea83961f5e677a71af
I20260924 10:07:36.296835 8509 ts_tablet_manager.cc:1918] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:07:36.297102 8509 log.cc:1196] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/17824395f88f46ea83961f5e677a71af
I20260924 10:07:36.297626 8509 ts_tablet_manager.cc:1939] T 17824395f88f46ea83961f5e677a71af P 97ce72fd9b574bdb9a37dd06cb75bace: Deleting consensus metadata
I20260924 10:07:36.297914 8508 ts_tablet_manager.cc:1918] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:07:36.298061 8508 log.cc:1196] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/17824395f88f46ea83961f5e677a71af
I20260924 10:07:36.298465 8508 ts_tablet_manager.cc:1939] T 17824395f88f46ea83961f5e677a71af P 151858e25ef04c8c8e0f37c93e1b7872: Deleting consensus metadata
I20260924 10:07:36.298560 8047 tablet_service.cc:1467] Processing CreateTablet for tablet ca1f4770d9434d83970bd105d598bdf3 (DEFAULT_TABLE table=test-table [id=5fda00d350e744be9fe9ce66529258aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:36.298960 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1f4770d9434d83970bd105d598bdf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:36.298954 7898 catalog_manager.cc:4841] TS 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193:39011): tablet 17824395f88f46ea83961f5e677a71af (table test-table [id=9d9df07eab0348c49b4631a25e1ab146]) successfully deleted
I20260924 10:07:36.299861 7896 catalog_manager.cc:4841] TS 151858e25ef04c8c8e0f37c93e1b7872 (127.6.2.194:34683): tablet 17824395f88f46ea83961f5e677a71af (table test-table [id=9d9df07eab0348c49b4631a25e1ab146]) successfully deleted
I20260924 10:07:36.299957 8510 ts_tablet_manager.cc:1939] T 17824395f88f46ea83961f5e677a71af P 307d18d9f9bb4d19baa09b4db2090b70: Deleting consensus metadata
I20260924 10:07:36.300685 8186 tablet_service.cc:1467] Processing CreateTablet for tablet ca1f4770d9434d83970bd105d598bdf3 (DEFAULT_TABLE table=test-table [id=5fda00d350e744be9fe9ce66529258aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:36.301055 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1f4770d9434d83970bd105d598bdf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:36.301882 7896 catalog_manager.cc:4841] TS 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195:42961): tablet 17824395f88f46ea83961f5e677a71af (table test-table [id=9d9df07eab0348c49b4631a25e1ab146]) successfully deleted
I20260924 10:07:36.302106 8325 tablet_service.cc:1467] Processing CreateTablet for tablet ca1f4770d9434d83970bd105d598bdf3 (DEFAULT_TABLE table=test-table [id=5fda00d350e744be9fe9ce66529258aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:36.302505 8325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1f4770d9434d83970bd105d598bdf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:36.307027 8520 tablet_bootstrap.cc:492] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872: Bootstrap starting.
I20260924 10:07:36.307235 8519 tablet_bootstrap.cc:492] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace: Bootstrap starting.
I20260924 10:07:36.308312 8521 tablet_bootstrap.cc:492] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70: Bootstrap starting.
I20260924 10:07:36.308730 8520 tablet_bootstrap.cc:654] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:36.308775 8519 tablet_bootstrap.cc:654] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:36.309836 8521 tablet_bootstrap.cc:654] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:36.311739 8521 tablet_bootstrap.cc:492] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70: No bootstrap required, opened a new log
I20260924 10:07:36.311776 8520 tablet_bootstrap.cc:492] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872: No bootstrap required, opened a new log
I20260924 10:07:36.311905 8521 ts_tablet_manager.cc:1397] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260924 10:07:36.311930 8520 ts_tablet_manager.cc:1397] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:07:36.312492 8519 tablet_bootstrap.cc:492] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace: No bootstrap required, opened a new log
I20260924 10:07:36.312731 8519 ts_tablet_manager.cc:1397] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260924 10:07:36.312636 8520 raft_consensus.cc:348] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.312630 8521 raft_consensus.cc:348] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.312882 8520 raft_consensus.cc:374] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:36.312883 8521 raft_consensus.cc:374] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:36.312968 8520 raft_consensus.cc:729] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151858e25ef04c8c8e0f37c93e1b7872, State: Initialized, Role: FOLLOWER
I20260924 10:07:36.312966 8521 raft_consensus.cc:729] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307d18d9f9bb4d19baa09b4db2090b70, State: Initialized, Role: FOLLOWER
I20260924 10:07:36.313207 8521 consensus_queue.cc:260] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.313253 8520 consensus_queue.cc:260] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.313716 8521 ts_tablet_manager.cc:1428] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:07:36.313887 8519 raft_consensus.cc:348] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.314185 8519 raft_consensus.cc:374] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:36.314309 8519 raft_consensus.cc:729] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ce72fd9b574bdb9a37dd06cb75bace, State: Initialized, Role: FOLLOWER
I20260924 10:07:36.314518 8520 ts_tablet_manager.cc:1428] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:07:36.314713 8519 consensus_queue.cc:260] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.315258 8519 ts_tablet_manager.cc:1428] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:07:36.504256 8543 raft_consensus.cc:482] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:07:36.504521 8543 raft_consensus.cc:504] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.505263 8543 leader_election.cc:288] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193:39011), 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195:42961)
I20260924 10:07:36.505774 8072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1f4770d9434d83970bd105d598bdf3" candidate_uuid: "151858e25ef04c8c8e0f37c93e1b7872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" is_pre_election: true
I20260924 10:07:36.505893 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1f4770d9434d83970bd105d598bdf3" candidate_uuid: "151858e25ef04c8c8e0f37c93e1b7872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307d18d9f9bb4d19baa09b4db2090b70" is_pre_election: true
I20260924 10:07:36.506124 8072 raft_consensus.cc:2454] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151858e25ef04c8c8e0f37c93e1b7872 in term 0.
I20260924 10:07:36.506157 8351 raft_consensus.cc:2454] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 151858e25ef04c8c8e0f37c93e1b7872 in term 0.
I20260924 10:07:36.506564 8139 leader_election.cc:302] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151858e25ef04c8c8e0f37c93e1b7872, 307d18d9f9bb4d19baa09b4db2090b70; no voters:
I20260924 10:07:36.506893 8543 raft_consensus.cc:2781] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:07:36.507009 8543 raft_consensus.cc:482] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:07:36.507128 8543 raft_consensus.cc:3037] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:36.509095 8543 raft_consensus.cc:504] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.509660 8543 leader_election.cc:288] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [CANDIDATE]: Term 1 election: Requested vote from peers 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193:39011), 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195:42961)
I20260924 10:07:36.510092 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1f4770d9434d83970bd105d598bdf3" candidate_uuid: "151858e25ef04c8c8e0f37c93e1b7872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307d18d9f9bb4d19baa09b4db2090b70"
I20260924 10:07:36.510123 8072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1f4770d9434d83970bd105d598bdf3" candidate_uuid: "151858e25ef04c8c8e0f37c93e1b7872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ce72fd9b574bdb9a37dd06cb75bace"
I20260924 10:07:36.510311 8351 raft_consensus.cc:3037] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:36.510370 8072 raft_consensus.cc:3037] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:36.512310 8351 raft_consensus.cc:2454] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151858e25ef04c8c8e0f37c93e1b7872 in term 1.
I20260924 10:07:36.512697 8139 leader_election.cc:302] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151858e25ef04c8c8e0f37c93e1b7872, 307d18d9f9bb4d19baa09b4db2090b70; no voters:
I20260924 10:07:36.512974 8543 raft_consensus.cc:2781] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:36.513136 8543 raft_consensus.cc:686] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 LEADER]: Becoming Leader. State: Replica: 151858e25ef04c8c8e0f37c93e1b7872, State: Running, Role: LEADER
I20260924 10:07:36.513612 8072 raft_consensus.cc:2454] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 151858e25ef04c8c8e0f37c93e1b7872 in term 1.
I20260924 10:07:36.513640 8543 consensus_queue.cc:237] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } }
I20260924 10:07:36.516433 7909 catalog_manager.cc:5495] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 reported cstate change: term changed from 0 to 1, leader changed from <none> to 151858e25ef04c8c8e0f37c93e1b7872 (127.6.2.194). New cstate: current_term: 1 leader_uuid: "151858e25ef04c8c8e0f37c93e1b7872" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "151858e25ef04c8c8e0f37c93e1b7872" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 } health_report { overall_health: UNKNOWN } } }
I20260924 10:07:36.566431 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:36.561216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59632 (local address 127.6.2.194:34683)
0924 10:07:36.561380 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:36.561385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:36.561622 (+ 237us) server_negotiation.cc:407] Connection header received
0924 10:07:36.561837 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:36.561843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:36.561914 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:36.562010 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:36.562422 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.563253 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.564541 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:36.565101 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:36.565233 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:36.565721 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:36.565821 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:36.565930 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:36.566101 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 10:07:36.566168 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:07:36.576138 8351 raft_consensus.cc:1261] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Refusing update from remote peer 151858e25ef04c8c8e0f37c93e1b7872: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:36.576216 8072 raft_consensus.cc:1261] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Refusing update from remote peer 151858e25ef04c8c8e0f37c93e1b7872: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:36.576643 8543 consensus_queue.cc:1035] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "307d18d9f9bb4d19baa09b4db2090b70" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:36.576908 8487 consensus_queue.cc:1035] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ce72fd9b574bdb9a37dd06cb75bace" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 39011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:07:36.600905 8485 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 10:07:38.184859 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:38.185132 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:38.189378 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 844ec8e9f4db407fb1077dfefdd615fc: Sending DeleteTablet for 3 replicas of tablet ca1f4770d9434d83970bd105d598bdf3
I20260924 10:07:38.190256 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1f4770d9434d83970bd105d598bdf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:38 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:46564
I20260924 10:07:38.190661 8187 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1f4770d9434d83970bd105d598bdf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:38 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:45864
I20260924 10:07:38.190914 8571 tablet_replica.cc:330] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace: stopping tablet replica
I20260924 10:07:38.191160 8571 raft_consensus.cc:2229] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:38.191213 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 7982
I20260924 10:07:38.191387 8572 tablet_replica.cc:330] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872: stopping tablet replica
I20260924 10:07:38.191437 8571 raft_consensus.cc:2258] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:38.191601 8572 raft_consensus.cc:2229] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:07:38.191929 8325 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1f4770d9434d83970bd105d598bdf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:38 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:49466
I20260924 10:07:38.191988 8572 raft_consensus.cc:2258] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:38.192528 8571 ts_tablet_manager.cc:1905] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:38.192580 8573 tablet_replica.cc:330] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70: stopping tablet replica
I20260924 10:07:38.192776 8573 raft_consensus.cc:2229] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:38.193495 8572 ts_tablet_manager.cc:1905] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:38.195297 8573 raft_consensus.cc:2258] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:38.196748 8573 ts_tablet_manager.cc:1905] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:38.199775 8572 ts_tablet_manager.cc:1918] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:07:38.199970 8572 log.cc:1196] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/ca1f4770d9434d83970bd105d598bdf3
I20260924 10:07:38.200549 8572 ts_tablet_manager.cc:1939] T ca1f4770d9434d83970bd105d598bdf3 P 151858e25ef04c8c8e0f37c93e1b7872: Deleting consensus metadata
I20260924 10:07:38.201727 7896 catalog_manager.cc:4841] TS 151858e25ef04c8c8e0f37c93e1b7872 (127.6.2.194:34683): tablet ca1f4770d9434d83970bd105d598bdf3 (table test-table [id=5fda00d350e744be9fe9ce66529258aa]) successfully deleted
I20260924 10:07:38.202761 8573 ts_tablet_manager.cc:1918] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:07:38.202934 8573 log.cc:1196] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/ca1f4770d9434d83970bd105d598bdf3
I20260924 10:07:38.203219 8571 ts_tablet_manager.cc:1918] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:07:38.203377 8571 log.cc:1196] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/ca1f4770d9434d83970bd105d598bdf3
I20260924 10:07:38.203399 8573 ts_tablet_manager.cc:1939] T ca1f4770d9434d83970bd105d598bdf3 P 307d18d9f9bb4d19baa09b4db2090b70: Deleting consensus metadata
I20260924 10:07:38.203920 8571 ts_tablet_manager.cc:1939] T ca1f4770d9434d83970bd105d598bdf3 P 97ce72fd9b574bdb9a37dd06cb75bace: Deleting consensus metadata
I20260924 10:07:38.204505 7896 catalog_manager.cc:4841] TS 307d18d9f9bb4d19baa09b4db2090b70 (127.6.2.195:42961): tablet ca1f4770d9434d83970bd105d598bdf3 (table test-table [id=5fda00d350e744be9fe9ce66529258aa]) successfully deleted
I20260924 10:07:38.204964 7898 catalog_manager.cc:4841] TS 97ce72fd9b574bdb9a37dd06cb75bace (127.6.2.193:39011): tablet ca1f4770d9434d83970bd105d598bdf3 (table test-table [id=5fda00d350e744be9fe9ce66529258aa]) successfully deleted
Sep 24 10:07:38 dist-test-slave-s3n3 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:07:38.218569 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:38.193034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.6.2.193:39011)
0924 10:07:38.203401 (+ 10367us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:38.203409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:38.203434 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:07:38.203517 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:38.203526 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:38.203634 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:38.203801 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:38.205501 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:38.206382 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:38.207835 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:38.208305 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:38.208415 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:38.212639 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:38.212655 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:38.212684 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:38.212731 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:38.215915 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:38.216541 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:38.216549 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:38.216552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:38.216692 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:38.217014 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:38.217018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:38.217020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:38.217436 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:38.217686 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:38.218094 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 10:07:38.218267 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10197,"thread_start_us":10934,"threads_started":1}
W20260924 10:07:38.219359 8113 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42920
W20260924 10:07:38.219991 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 7982: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:07:38.220227 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 7982
I20260924 10:07:38.242489 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 8122
Sep 24 10:07:38 dist-test-slave-s3n3 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:07:38.263712 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:38.243938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.6.2.194:34683)
0924 10:07:38.244528 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:38.244532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:38.244550 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:07:38.244603 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:38.244607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:38.244665 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:38.244888 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:38.246150 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:38.246905 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:38.248217 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:38.248635 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:38.248797 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:38.258070 (+ 9273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:38.258088 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:38.258104 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:38.258145 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:38.260961 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:38.261634 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:38.261647 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:38.261650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:38.261765 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:38.262120 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:38.262127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:38.262131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:38.262494 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:38.262785 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:38.263134 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 10:07:38.263365 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":294,"threads_started":1}
W20260924 10:07:38.264410 8252 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59642
W20260924 10:07:38.265257 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 8122: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:07:38.265396 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 8122
I20260924 10:07:38.291683 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 8261
Sep 24 10:07:38 dist-test-slave-s3n3 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:07:38.306896 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:38.292556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.6.2.195:42961)
0924 10:07:38.293103 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:38.293108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:38.293124 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:07:38.293175 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:38.293179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:38.293235 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:38.293329 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:38.294579 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:38.295421 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:38.296632 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:38.297007 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:38.297101 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:38.301416 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:38.301429 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:38.301445 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:38.301480 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:38.304180 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:38.304847 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:38.304862 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:38.304864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:38.304987 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:38.305353 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:38.305360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:38.305363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:38.305710 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:38.306028 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:38.306320 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 10:07:38.306543 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":307,"threads_started":1}
W20260924 10:07:38.307580 8391 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39960
W20260924 10:07:38.308190 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 8261: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:07:38.308372 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 8261
I20260924 10:07:38.330292 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 7879
I20260924 10:07:38.331537 7951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:38.447196 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 7879
2026-09-24T10:07:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8465 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:07:38 dist-test-slave-s3n3 krb5kdc[8581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:07:38 dist-test-slave-s3n3 krb5kdc[8581](info): set up 2 sockets
Sep 24 10:07:38 dist-test-slave-s3n3 krb5kdc[8581](info): commencing operation
krb5kdc: starting...
W20260924 10:07:39.543519 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:07:39 dist-test-slave-s3n3 krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:07:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:07:39.747526 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:41121
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:41121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:40.069100 8594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:40.069406 8594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:40.069495 8594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:40.076833 8594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:40.076978 8594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:40.077025 8594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:40.077075 8594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:40.077119 8594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:40.087119 8594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:41121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:41121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:40.088388 8594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:40.089607 8594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:40.101935 8600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:40.102000 8599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:40.102000 8602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:40.102638 8594 server_base.cc:1034] running on GCE node
I20260924 10:07:40.103526 8594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:40.105067 8594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:40.106331 8594 hybrid_clock.cc:648] HybridClock initialized: now 1790244460106261 us; error 55 us; skew 500 ppm
I20260924 10:07:40.106683 8594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:40 dist-test-slave-s3n3 krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244460, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:40.109885 8594 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:07:40.111480 8594 webserver.cc:460] Webserver started at http://127.6.2.254:37381/ using document root <none> and password file <none>
I20260924 10:07:40.112169 8594 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:40.112317 8594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:40.112623 8594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:40.115754 8594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "d47f98650cf14f7a85e5fb04c67e9d67"
format_stamp: "Formatted at 2026-09-24 10:07:40 on dist-test-slave-s3n3"
server_key: "366affd4f1ebb0b0fbc4781e1548618f"
server_key_iv: "8531a0108bcfd10b1e82583f4955f64d"
server_key_version: "encryptionkey@0"
I20260924 10:07:40.116528 8594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "d47f98650cf14f7a85e5fb04c67e9d67"
format_stamp: "Formatted at 2026-09-24 10:07:40 on dist-test-slave-s3n3"
server_key: "366affd4f1ebb0b0fbc4781e1548618f"
server_key_iv: "8531a0108bcfd10b1e82583f4955f64d"
server_key_version: "encryptionkey@0"
I20260924 10:07:40.121799 8594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:07:40.125512 8609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:40.127226 8594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:07:40.127426 8594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "d47f98650cf14f7a85e5fb04c67e9d67"
format_stamp: "Formatted at 2026-09-24 10:07:40 on dist-test-slave-s3n3"
server_key: "366affd4f1ebb0b0fbc4781e1548618f"
server_key_iv: "8531a0108bcfd10b1e82583f4955f64d"
server_key_version: "encryptionkey@0"
I20260924 10:07:40.127637 8594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:40.159022 8594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:40.163937 8594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:40.164249 8594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:40.185217 8594 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:41121
I20260924 10:07:40.185247 8660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:41121 every 8 connection(s)
I20260924 10:07:40.186620 8594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:40.190410 8661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:40.196807 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 8594
I20260924 10:07:40.197057 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:07:40.197865 6155 external_mini_cluster.cc:1428] Setting key 1c40d5fedbc19a9ad1ee52343f624ba5
I20260924 10:07:40.201767 8661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67: Bootstrap starting.
I20260924 10:07:40.205523 8661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:40.206565 8661 log.cc:824] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:40.209753 8661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67: No bootstrap required, opened a new log
Sep 24 10:07:40 dist-test-slave-s3n3 krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:40.217660 8661 raft_consensus.cc:348] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f98650cf14f7a85e5fb04c67e9d67" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41121 } }
I20260924 10:07:40.218089 8661 raft_consensus.cc:374] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:40.218230 8661 raft_consensus.cc:729] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47f98650cf14f7a85e5fb04c67e9d67, State: Initialized, Role: FOLLOWER
I20260924 10:07:40.218951 8661 consensus_queue.cc:260] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f98650cf14f7a85e5fb04c67e9d67" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41121 } }
I20260924 10:07:40.219334 8661 raft_consensus.cc:388] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:40.219514 8661 raft_consensus.cc:482] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:40.219708 8661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:40.222029 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:40.200029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.6.2.254:41121)
0924 10:07:40.200862 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:40.200873 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:40.200911 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:07:40.202659 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:40.202681 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:40.203125 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:40.203457 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:40.205198 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:40.206588 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:40.208348 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:40.209005 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:40.209148 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:40.213609 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:40.213621 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:40.213636 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:40.213677 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:40.216965 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:40.217709 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:40.217716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:40.217723 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:40.217904 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:40.218411 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:40.218415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:40.218419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:40.219282 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:40.219624 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:40.220061 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 10:07:40.220541 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":424,"threads_started":1}
I20260924 10:07:40.222956 8661 raft_consensus.cc:504] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f98650cf14f7a85e5fb04c67e9d67" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41121 } }
I20260924 10:07:40.223568 8661 leader_election.cc:302] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d47f98650cf14f7a85e5fb04c67e9d67; no voters:
I20260924 10:07:40.224332 8661 leader_election.cc:288] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:40.224462 8666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:40.225960 8666 raft_consensus.cc:686] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [term 1 LEADER]: Becoming Leader. State: Replica: d47f98650cf14f7a85e5fb04c67e9d67, State: Running, Role: LEADER
I20260924 10:07:40.226600 8661 sys_catalog.cc:564] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:40.226500 8666 consensus_queue.cc:237] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f98650cf14f7a85e5fb04c67e9d67" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41121 } }
I20260924 10:07:40.229074 8668 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d47f98650cf14f7a85e5fb04c67e9d67. Latest consensus state: current_term: 1 leader_uuid: "d47f98650cf14f7a85e5fb04c67e9d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f98650cf14f7a85e5fb04c67e9d67" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41121 } } }
I20260924 10:07:40.229275 8667 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d47f98650cf14f7a85e5fb04c67e9d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f98650cf14f7a85e5fb04c67e9d67" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41121 } } }
I20260924 10:07:40.229518 8668 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:40.229559 8667 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:40.235052 8673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:40.240481 8673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:40.249249 8673 catalog_manager.cc:1347] Generated new cluster ID: 4dc57cdf38b84c10b48346d5aed294c0
I20260924 10:07:40.249374 8673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:40.267843 8673 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:40.269255 8673 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:40.279491 8673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d47f98650cf14f7a85e5fb04c67e9d67: Generated new TSK 0
I20260924 10:07:40.280378 8673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:07:40.331315 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41121
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:40.675894 8687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:40.676187 8687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:40.676278 8687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:40.683578 8687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:40.683696 8687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:40.683809 8687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:07:40.693645 8687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:40.694902 8687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:40.696230 8687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:40.711201 8692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:40.711375 8693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:40.711406 8695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:40.712779 8687 server_base.cc:1034] running on GCE node
I20260924 10:07:40.713490 8687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:40.714704 8687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:40.716149 8687 hybrid_clock.cc:648] HybridClock initialized: now 1790244460716084 us; error 72 us; skew 500 ppm
I20260924 10:07:40.716547 8687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:40 dist-test-slave-s3n3 krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244460, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:40.720022 8687 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:07:40.721879 8687 webserver.cc:460] Webserver started at http://127.6.2.193:42937/ using document root <none> and password file <none>
I20260924 10:07:40.722563 8687 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:40.722663 8687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:40.722970 8687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:40.725869 8687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "b56b66f1c0504e9a92e4cb864bf746de"
format_stamp: "Formatted at 2026-09-24 10:07:40 on dist-test-slave-s3n3"
server_key: "8cffaf45ff008e88dfe08b52c38b2e50"
server_key_iv: "287f62b280e550dce188711cbd5690d9"
server_key_version: "encryptionkey@0"
I20260924 10:07:40.726702 8687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "b56b66f1c0504e9a92e4cb864bf746de"
format_stamp: "Formatted at 2026-09-24 10:07:40 on dist-test-slave-s3n3"
server_key: "8cffaf45ff008e88dfe08b52c38b2e50"
server_key_iv: "287f62b280e550dce188711cbd5690d9"
server_key_version: "encryptionkey@0"
I20260924 10:07:40.732117 8687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 10:07:40.735877 8702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:40.737339 8687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:40.737481 8687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "b56b66f1c0504e9a92e4cb864bf746de"
format_stamp: "Formatted at 2026-09-24 10:07:40 on dist-test-slave-s3n3"
server_key: "8cffaf45ff008e88dfe08b52c38b2e50"
server_key_iv: "287f62b280e550dce188711cbd5690d9"
server_key_version: "encryptionkey@0"
I20260924 10:07:40.737653 8687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:40.772315 8687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:40.776723 8687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:40.776968 8687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:40.777907 8687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:40.779508 8687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:40.779584 8687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:40.779733 8687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:40.779807 8687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:40.813859 8687 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:44343
I20260924 10:07:40.813941 8814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:44343 every 8 connection(s)
I20260924 10:07:40.815250 8687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:07:40.823724 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 8687
I20260924 10:07:40.823968 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:07:40.824568 6155 external_mini_cluster.cc:1428] Setting key a6d5856fd52aa4a2f5caa178e9a1047a
Sep 24 10:07:40 dist-test-slave-s3n3 krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244460, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:40.837742 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:40.817862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:49583 (local address 127.6.2.254:41121)
0924 10:07:40.818335 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:40.818343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:40.819285 (+ 942us) server_negotiation.cc:407] Connection header received
0924 10:07:40.821016 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:40.821021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:40.821094 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:40.821222 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:40.822941 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:40.823826 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:40.825690 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:40.826227 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:40.826396 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:40.830353 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:40.830362 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:40.830369 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:40.830416 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:40.834164 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:40.835119 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:40.835127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:40.835134 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:40.835281 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:40.835687 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:40.835695 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:40.835701 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:40.836108 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:40.836251 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:40.836968 (+ 717us) server_negotiation.cc:299] Negotiation successful
0924 10:07:40.837246 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":338,"thread_start_us":213,"threads_started":1}
I20260924 10:07:40.838518 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:40.818397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41121 (local address 127.6.2.193:49583)
0924 10:07:40.819133 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:40.819176 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:40.820741 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:40.821460 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:40.821471 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:40.822012 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:40.822741 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:40.822763 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:40.823997 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:40.824009 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:40.825529 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:40.825545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:40.826553 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:40.826563 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:40.826801 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:40.827624 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:40.827651 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:40.830090 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:40.834370 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:40.834384 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:40.834408 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:40.834951 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:40.835393 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:40.835396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:40.835399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:40.835571 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:40.836238 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:40.836245 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:40.836585 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 10:07:40.836819 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 10:07:40.837244 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":267,"threads_started":1}
I20260924 10:07:40.840152 8815 heartbeater.cc:347] Connected to a master server at 127.6.2.254:41121
I20260924 10:07:40.840523 8815 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:40.841544 8815 heartbeater.cc:511] Master 127.6.2.254:41121 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
I20260924 10:07:40.843950 8626 ts_manager.cc:194] Registered new tserver with Master: b56b66f1c0504e9a92e4cb864bf746de (127.6.2.193:44343)
I20260924 10:07:40.845410 8626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:49583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:07:40.871112 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41121
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:41.194403 8821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:41.194677 8821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:41.194835 8821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:41.202292 8821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:41.202414 8821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:41.202526 8821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:07:41.212781 8821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:41.214107 8821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:41.215619 8821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:41.229626 8826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:41.229540 8827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:41.229576 8829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:41.230530 8821 server_base.cc:1034] running on GCE node
I20260924 10:07:41.231271 8821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:41.232327 8821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:41.233613 8821 hybrid_clock.cc:648] HybridClock initialized: now 1790244461233533 us; error 59 us; skew 500 ppm
I20260924 10:07:41.233980 8821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:41 dist-test-slave-s3n3 krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244461, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:41.237439 8821 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:07:41.239167 8821 webserver.cc:460] Webserver started at http://127.6.2.194:43075/ using document root <none> and password file <none>
I20260924 10:07:41.239888 8821 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:41.240037 8821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:41.240360 8821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:41.243507 8821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "5e3f8e8f1ba1477da242b59a5fafd25d"
format_stamp: "Formatted at 2026-09-24 10:07:41 on dist-test-slave-s3n3"
server_key: "4cc11e78eb41a577a71cb67fef0650af"
server_key_iv: "c662ee245ffb8bd6061a60d5cd88a582"
server_key_version: "encryptionkey@0"
I20260924 10:07:41.244302 8821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "5e3f8e8f1ba1477da242b59a5fafd25d"
format_stamp: "Formatted at 2026-09-24 10:07:41 on dist-test-slave-s3n3"
server_key: "4cc11e78eb41a577a71cb67fef0650af"
server_key_iv: "c662ee245ffb8bd6061a60d5cd88a582"
server_key_version: "encryptionkey@0"
I20260924 10:07:41.249787 8821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:07:41.253499 8836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:41.255194 8821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:07:41.255371 8821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "5e3f8e8f1ba1477da242b59a5fafd25d"
format_stamp: "Formatted at 2026-09-24 10:07:41 on dist-test-slave-s3n3"
server_key: "4cc11e78eb41a577a71cb67fef0650af"
server_key_iv: "c662ee245ffb8bd6061a60d5cd88a582"
server_key_version: "encryptionkey@0"
I20260924 10:07:41.255587 8821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:41.273149 8821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:41.277468 8821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:41.277783 8821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:41.278807 8821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:41.280391 8821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:41.280496 8821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:41.280648 8821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:41.280720 8821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:41.314831 8821 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:41785
I20260924 10:07:41.314869 8948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:41785 every 8 connection(s)
I20260924 10:07:41.316129 8821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:07:41.320353 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 8821
I20260924 10:07:41.320591 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:07:41.321151 6155 external_mini_cluster.cc:1428] Setting key 66eb3452c16b8f5d8d369c55c52c7a85
Sep 24 10:07:41 dist-test-slave-s3n3 krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244461, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:07:41.340785 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:41.319182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:60555 (local address 127.6.2.254:41121)
0924 10:07:41.319482 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:41.319491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:41.321348 (+ 1857us) server_negotiation.cc:407] Connection header received
0924 10:07:41.323693 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:41.323698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:41.323765 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:41.323860 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:41.326042 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:41.326854 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:41.329298 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:41.329714 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:41.329861 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:41.333598 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:41.333607 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:41.333614 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:41.333660 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:41.337219 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:41.338268 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:41.338272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:41.338274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:41.338389 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:41.338898 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:41.338901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:41.338907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:41.339227 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:41.339378 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:41.340187 (+ 809us) server_negotiation.cc:299] Negotiation successful
0924 10:07:41.340419 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 10:07:41.341892 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:41.320048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41121 (local address 127.6.2.194:60555)
0924 10:07:41.321125 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:41.321192 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:41.323393 (+ 2201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:41.324152 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:41.324168 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:41.325001 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:41.325850 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:41.325878 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:41.327001 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:41.327010 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:41.329138 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:41.329151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:41.329941 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:41.329948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:41.330182 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:41.330874 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:41.330896 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:41.333346 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:41.337386 (+ 4040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:41.337400 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:41.337423 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:41.338135 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:41.338502 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:41.338509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:41.338513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:41.338775 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:41.339356 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:41.339364 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:41.339756 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 10:07:41.340102 (+ 346us) client_negotiation.cc:240] Negotiation successful
0924 10:07:41.340655 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":513,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
I20260924 10:07:41.343786 8949 heartbeater.cc:347] Connected to a master server at 127.6.2.254:41121
I20260924 10:07:41.344111 8949 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:41.344832 8949 heartbeater.cc:511] Master 127.6.2.254:41121 requested a full tablet report, sending...
I20260924 10:07:41.346321 8626 ts_manager.cc:194] Registered new tserver with Master: 5e3f8e8f1ba1477da242b59a5fafd25d (127.6.2.194:41785)
I20260924 10:07:41.347221 8626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:60555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:07:41.367815 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41121
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:41.712334 8954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:41.712633 8954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:41.712742 8954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:41.720228 8954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:41.720363 8954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:41.720481 8954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:07:41.731329 8954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:41.732825 8954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:41.734354 8954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:41.748497 8960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:41.748783 8962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:41.748514 8959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:41.749157 8954 server_base.cc:1034] running on GCE node
I20260924 10:07:41.749842 8954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:41.750975 8954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:41.752207 8954 hybrid_clock.cc:648] HybridClock initialized: now 1790244461752170 us; error 37 us; skew 500 ppm
I20260924 10:07:41.752534 8954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:41 dist-test-slave-s3n3 krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244461, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:41.755810 8954 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:07:41.757484 8954 webserver.cc:460] Webserver started at http://127.6.2.195:44985/ using document root <none> and password file <none>
I20260924 10:07:41.758143 8954 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:41.758241 8954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:41.758489 8954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:41.761648 8954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "a06427c113d84ca1a104bf841d80443b"
format_stamp: "Formatted at 2026-09-24 10:07:41 on dist-test-slave-s3n3"
server_key: "be582a84a96507e671631ff60be01c75"
server_key_iv: "2be0aa23286fe6416869d60b0cb022f0"
server_key_version: "encryptionkey@0"
I20260924 10:07:41.762404 8954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "a06427c113d84ca1a104bf841d80443b"
format_stamp: "Formatted at 2026-09-24 10:07:41 on dist-test-slave-s3n3"
server_key: "be582a84a96507e671631ff60be01c75"
server_key_iv: "2be0aa23286fe6416869d60b0cb022f0"
server_key_version: "encryptionkey@0"
I20260924 10:07:41.768150 8954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 10:07:41.771754 8969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:41.773422 8954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:41.773635 8954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "a06427c113d84ca1a104bf841d80443b"
format_stamp: "Formatted at 2026-09-24 10:07:41 on dist-test-slave-s3n3"
server_key: "be582a84a96507e671631ff60be01c75"
server_key_iv: "2be0aa23286fe6416869d60b0cb022f0"
server_key_version: "encryptionkey@0"
I20260924 10:07:41.773866 8954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:41.790494 8954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:41.795204 8954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:41.795526 8954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:41.796581 8954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:41.798096 8954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:41.798170 8954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:41.798301 8954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:41.798369 8954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:41.834633 8954 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:40465
I20260924 10:07:41.834679 9081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:40465 every 8 connection(s)
I20260924 10:07:41.835988 8954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:07:41.838455 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 8954
I20260924 10:07:41.838660 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:07:41.839257 6155 external_mini_cluster.cc:1428] Setting key 947200ae834f2dcc5b4935dc21ca365f
I20260924 10:07:41.848875 8815 heartbeater.cc:503] Master 127.6.2.254:41121 was elected leader, sending a full tablet report...
Sep 24 10:07:41 dist-test-slave-s3n3 krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244461, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:41.859483 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:41.838809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:55837 (local address 127.6.2.254:41121)
0924 10:07:41.838998 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:41.839002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:41.840616 (+ 1614us) server_negotiation.cc:407] Connection header received
0924 10:07:41.843138 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:41.843150 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:41.843223 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:41.843321 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:41.844837 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:41.845610 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:41.847565 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:41.847973 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:41.848089 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:41.851951 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:41.851956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:41.851959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:41.851993 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:41.854966 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:41.856547 (+ 1581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:41.856553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:41.856556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:41.856719 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:41.857193 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:41.857197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:41.857199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:41.857560 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:41.857679 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:41.858680 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0924 10:07:41.858943 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:07:41.860316 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:41.839502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41121 (local address 127.6.2.195:55837)
0924 10:07:41.840406 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:41.840476 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:41.842808 (+ 2332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:41.843546 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:41.843554 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:41.844092 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:41.844644 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:41.844660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:41.845757 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:41.845761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:41.847042 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:41.847053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:41.848224 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:41.848230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:41.848436 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:41.849222 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:41.849245 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:41.851729 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:41.855321 (+ 3592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:41.855438 (+ 117us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:41.855470 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:41.856359 (+ 889us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:41.856844 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:41.856850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:41.856853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:41.857046 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:41.857694 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:41.857700 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:41.858075 (+ 375us) client_negotiation.cc:769] Sending connection context
0924 10:07:41.858325 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 10:07:41.858905 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":375,"threads_started":1}
I20260924 10:07:41.861989 9082 heartbeater.cc:347] Connected to a master server at 127.6.2.254:41121
I20260924 10:07:41.862354 9082 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:41.863205 9082 heartbeater.cc:511] Master 127.6.2.254:41121 requested a full tablet report, sending...
I20260924 10:07:41.864975 8626 ts_manager.cc:194] Registered new tserver with Master: a06427c113d84ca1a104bf841d80443b (127.6.2.195:40465)
I20260924 10:07:41.865779 8626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:55837
I20260924 10:07:41.869534 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:07:41.903616 8817 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:07:41.901037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.6.2.254:41121)
0924 10:07:41.901333 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:41.901341 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:41.901640 (+ 299us) server_negotiation.cc:407] Connection header received
0924 10:07:41.901793 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:41.901796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:41.901863 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:41.901990 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:41.903337 (+ 1347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50892: BlockingRecv error: recv got EOF from 127.0.0.1:50892 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 10:07:41.906630 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 8687
W20260924 10:07:41.909868 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 8687: Not authorized: Client connection negotiation failed: client connection to 127.6.2.193:44343: server requires authentication, but client does not have Kerberos credentials available
I20260924 10:07:41.910080 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 8687
I20260924 10:07:41.926839 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 8821
W20260924 10:07:41.929697 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 8821: Not authorized: Client connection negotiation failed: client connection to 127.6.2.194:41785: server requires authentication, but client does not have Kerberos credentials available
I20260924 10:07:41.929880 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 8821
I20260924 10:07:41.946354 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 8954
W20260924 10:07:41.949417 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 8954: Not authorized: Client connection negotiation failed: client connection to 127.6.2.195:40465: server requires authentication, but client does not have Kerberos credentials available
I20260924 10:07:41.949569 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 8954
I20260924 10:07:41.964959 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 8594
I20260924 10:07:41.966176 8656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:42.069916 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 8594
2026-09-24T10:07:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3621 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:07:42 dist-test-slave-s3n3 krb5kdc[9099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:07:42 dist-test-slave-s3n3 krb5kdc[9099](info): set up 2 sockets
Sep 24 10:07:42 dist-test-slave-s3n3 krb5kdc[9099](info): commencing operation
krb5kdc: starting...
W20260924 10:07:43.175529 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:07:43 dist-test-slave-s3n3 krb5kdc[9099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:07:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:07:43.384646 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:40041
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:40041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:43.713469 9112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:43.713797 9112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:43.713889 9112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:43.721215 9112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:43.721341 9112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:43.721387 9112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:43.721436 9112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:43.721490 9112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:43.731309 9112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:40041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:40041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:43.732595 9112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:43.733908 9112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:43.745885 9117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:43.746009 9120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:43.745887 9118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:43.746663 9112 server_base.cc:1034] running on GCE node
I20260924 10:07:43.747511 9112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:43.748974 9112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:43.750211 9112 hybrid_clock.cc:648] HybridClock initialized: now 1790244463750170 us; error 39 us; skew 500 ppm
I20260924 10:07:43.750573 9112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:43 dist-test-slave-s3n3 krb5kdc[9099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244463, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:43.762276 9112 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:07:43.763996 9112 webserver.cc:460] Webserver started at http://127.6.2.254:33325/ using document root <none> and password file <none>
I20260924 10:07:43.764776 9112 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:43.764932 9112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:43.765247 9112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:43.768795 9112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "0dcd2ee6db0645e399080739d9c0f777"
format_stamp: "Formatted at 2026-09-24 10:07:43 on dist-test-slave-s3n3"
server_key: "61f72fb3a6c4420197e9c21cdb75fd6b"
server_key_iv: "dc36f8fe30b6d942da3d1d39c1df6890"
server_key_version: "encryptionkey@0"
I20260924 10:07:43.769673 9112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "0dcd2ee6db0645e399080739d9c0f777"
format_stamp: "Formatted at 2026-09-24 10:07:43 on dist-test-slave-s3n3"
server_key: "61f72fb3a6c4420197e9c21cdb75fd6b"
server_key_iv: "dc36f8fe30b6d942da3d1d39c1df6890"
server_key_version: "encryptionkey@0"
I20260924 10:07:43.776070 9112 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 10:07:43.781457 9127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:43.783820 9112 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:07:43.784301 9112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "0dcd2ee6db0645e399080739d9c0f777"
format_stamp: "Formatted at 2026-09-24 10:07:43 on dist-test-slave-s3n3"
server_key: "61f72fb3a6c4420197e9c21cdb75fd6b"
server_key_iv: "dc36f8fe30b6d942da3d1d39c1df6890"
server_key_version: "encryptionkey@0"
I20260924 10:07:43.784607 9112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:43.809065 9112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:43.813952 9112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:43.814213 9112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:43.835542 9112 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:40041
I20260924 10:07:43.835577 9178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:40041 every 8 connection(s)
I20260924 10:07:43.836987 9112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:43.840790 9179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:43.846776 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 9112
I20260924 10:07:43.846997 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:07:43.847666 6155 external_mini_cluster.cc:1428] Setting key 4bdd05998cee682bbdc3e836f15fd741
I20260924 10:07:43.851812 9179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777: Bootstrap starting.
I20260924 10:07:43.855657 9179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:43.856791 9179 log.cc:824] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:43.859643 9179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777: No bootstrap required, opened a new log
Sep 24 10:07:43 dist-test-slave-s3n3 krb5kdc[9099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:43.868342 9179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcd2ee6db0645e399080739d9c0f777" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40041 } }
I20260924 10:07:43.868747 9179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:43.868877 9179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dcd2ee6db0645e399080739d9c0f777, State: Initialized, Role: FOLLOWER
I20260924 10:07:43.869585 9179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcd2ee6db0645e399080739d9c0f777" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40041 } }
I20260924 10:07:43.869822 9179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:43.869958 9179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:43.870136 9179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:43.872642 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:43.850166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47718 (local address 127.6.2.254:40041)
0924 10:07:43.850989 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:43.851001 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:43.851043 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:07:43.852875 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:43.852897 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:43.853301 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:43.853627 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:43.855867 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:43.857335 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:43.858661 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:43.859432 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:43.859623 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:43.863993 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:43.864002 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:43.864020 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:43.864072 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:43.867936 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:43.868795 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:43.868802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:43.868809 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:43.869075 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:43.869500 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:43.869505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:43.869508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:43.870284 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:43.870593 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:43.871138 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 10:07:43.871597 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":363,"threads_started":1}
I20260924 10:07:43.872727 9179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcd2ee6db0645e399080739d9c0f777" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40041 } }
I20260924 10:07:43.873381 9179 leader_election.cc:302] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dcd2ee6db0645e399080739d9c0f777; no voters:
I20260924 10:07:43.874064 9179 leader_election.cc:288] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:43.874260 9184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:43.875501 9184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [term 1 LEADER]: Becoming Leader. State: Replica: 0dcd2ee6db0645e399080739d9c0f777, State: Running, Role: LEADER
I20260924 10:07:43.876123 9184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcd2ee6db0645e399080739d9c0f777" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40041 } }
I20260924 10:07:43.876511 9179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:43.877197 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:07:43.879022 9186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dcd2ee6db0645e399080739d9c0f777. Latest consensus state: current_term: 1 leader_uuid: "0dcd2ee6db0645e399080739d9c0f777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcd2ee6db0645e399080739d9c0f777" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40041 } } }
I20260924 10:07:43.880216 9186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:43.881507 9190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:43.882000 9185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dcd2ee6db0645e399080739d9c0f777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcd2ee6db0645e399080739d9c0f777" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40041 } } }
I20260924 10:07:43.882498 9185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:43.888265 9190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:43.909008 9190 catalog_manager.cc:1347] Generated new cluster ID: 593365bca2dc4e228be4001b2f646924
I20260924 10:07:43.909164 9190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:43.914149 9204 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:07:43.933961 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:43.924685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47730 (local address 127.6.2.254:40041)
0924 10:07:43.924873 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:43.924879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:43.925436 (+ 557us) server_negotiation.cc:407] Connection header received
0924 10:07:43.925612 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:43.925618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:43.925707 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:43.925830 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:43.927010 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:43.928186 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:43.929985 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:43.930642 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:43.930743 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:43.932166 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:43.932173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:43.932178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:43.933020 (+ 842us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:43.933079 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:43.933105 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:43.933181 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:43.933447 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 10:07:43.933561 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:07:43.964334 9190 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:43.966245 9190 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:43.975592 9190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dcd2ee6db0645e399080739d9c0f777: Generated new TSK 0
I20260924 10:07:43.976709 9190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:07:43.985692 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 9112
I20260924 10:07:43.987831 9174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:44.082556 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 9112
2026-09-24T10:07:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2007 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:07:44 dist-test-slave-s3n3 krb5kdc[9213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:07:44 dist-test-slave-s3n3 krb5kdc[9213](info): set up 2 sockets
Sep 24 10:07:44 dist-test-slave-s3n3 krb5kdc[9213](info): commencing operation
krb5kdc: starting...
W20260924 10:07:45.187526 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:07:45 dist-test-slave-s3n3 krb5kdc[9213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:07:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:07:45.388262 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:38099
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:38099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:45.716456 9226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:45.716778 9226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:45.716948 9226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:45.724749 9226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:45.724874 9226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:45.724958 9226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:45.725067 9226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:45.725128 9226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:45.735481 9226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:38099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:38099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:45.736982 9226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:45.738360 9226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:45.752105 9234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:45.752075 9231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:45.752346 9232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:45.753144 9226 server_base.cc:1034] running on GCE node
I20260924 10:07:45.754014 9226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:45.755589 9226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:45.756819 9226 hybrid_clock.cc:648] HybridClock initialized: now 1790244465756765 us; error 38 us; skew 500 ppm
I20260924 10:07:45.757225 9226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:45 dist-test-slave-s3n3 krb5kdc[9213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244465, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:45.760628 9226 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:07:45.762264 9226 webserver.cc:460] Webserver started at http://127.6.2.254:40129/ using document root <none> and password file <none>
I20260924 10:07:45.762988 9226 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:45.763180 9226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:45.763501 9226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:45.766707 9226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "cb48b52d795545cda5deb42a5f0f26f6"
format_stamp: "Formatted at 2026-09-24 10:07:45 on dist-test-slave-s3n3"
server_key: "4c092275e578418d951d954623d4e89a"
server_key_iv: "ec6f2b6b4070ca830017f291266e44ab"
server_key_version: "encryptionkey@0"
I20260924 10:07:45.767539 9226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "cb48b52d795545cda5deb42a5f0f26f6"
format_stamp: "Formatted at 2026-09-24 10:07:45 on dist-test-slave-s3n3"
server_key: "4c092275e578418d951d954623d4e89a"
server_key_iv: "ec6f2b6b4070ca830017f291266e44ab"
server_key_version: "encryptionkey@0"
I20260924 10:07:45.773288 9226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 10:07:45.776875 9241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:45.778391 9226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:07:45.778626 9226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "cb48b52d795545cda5deb42a5f0f26f6"
format_stamp: "Formatted at 2026-09-24 10:07:45 on dist-test-slave-s3n3"
server_key: "4c092275e578418d951d954623d4e89a"
server_key_iv: "ec6f2b6b4070ca830017f291266e44ab"
server_key_version: "encryptionkey@0"
I20260924 10:07:45.778844 9226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:45.802769 9226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:45.807353 9226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:45.807686 9226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:45.828794 9226 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:38099
I20260924 10:07:45.828842 9292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:38099 every 8 connection(s)
I20260924 10:07:45.830191 9226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:45.834458 9293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:45.838737 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 9226
I20260924 10:07:45.838963 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:07:45.839644 6155 external_mini_cluster.cc:1428] Setting key 6623085fcf526ba7bf37bf6c09fec2b0
I20260924 10:07:45.847572 9293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6: Bootstrap starting.
I20260924 10:07:45.852035 9293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:45.853327 9293 log.cc:824] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:45.856603 9293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6: No bootstrap required, opened a new log
Sep 24 10:07:45 dist-test-slave-s3n3 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:45.864840 9293 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb48b52d795545cda5deb42a5f0f26f6" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38099 } }
I20260924 10:07:45.865341 9293 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:45.865458 9293 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb48b52d795545cda5deb42a5f0f26f6, State: Initialized, Role: FOLLOWER
I20260924 10:07:45.866150 9293 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb48b52d795545cda5deb42a5f0f26f6" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38099 } }
I20260924 10:07:45.866297 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:45.841795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.6.2.254:38099)
0924 10:07:45.842664 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:45.842679 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:45.842724 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:07:45.845358 (+ 2634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:45.845383 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:45.845793 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:45.846147 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:45.848430 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:45.849813 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:45.852305 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:45.853561 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:45.854002 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:45.858482 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:45.858490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:45.858510 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:45.858547 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:45.861955 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:45.862714 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:45.862720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:45.862726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:45.862888 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:45.863352 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:45.863356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:45.863358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:45.863915 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:45.864254 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:45.864693 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 10:07:45.865118 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":392,"threads_started":1}
I20260924 10:07:45.866408 9293 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:45.866566 9293 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:45.866735 9293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:45.869520 9293 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb48b52d795545cda5deb42a5f0f26f6" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38099 } }
I20260924 10:07:45.869933 9293 leader_election.cc:302] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb48b52d795545cda5deb42a5f0f26f6; no voters:
I20260924 10:07:45.870440 9293 leader_election.cc:288] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:45.870580 9298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:45.871289 9298 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [term 1 LEADER]: Becoming Leader. State: Replica: cb48b52d795545cda5deb42a5f0f26f6, State: Running, Role: LEADER
I20260924 10:07:45.871866 9298 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb48b52d795545cda5deb42a5f0f26f6" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38099 } }
I20260924 10:07:45.872457 9293 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:45.874351 9300 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb48b52d795545cda5deb42a5f0f26f6. Latest consensus state: current_term: 1 leader_uuid: "cb48b52d795545cda5deb42a5f0f26f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb48b52d795545cda5deb42a5f0f26f6" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38099 } } }
I20260924 10:07:45.874698 9300 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:45.875536 9303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:45.875490 9299 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb48b52d795545cda5deb42a5f0f26f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb48b52d795545cda5deb42a5f0f26f6" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38099 } } }
I20260924 10:07:45.875882 9299 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:45.881659 9303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:45.894009 9303 catalog_manager.cc:1347] Generated new cluster ID: 2dcc940465a84a47ab70350748c7000d
I20260924 10:07:45.894146 9303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:45.912278 9303 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:45.914245 9303 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:45.925203 9303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb48b52d795545cda5deb42a5f0f26f6: Generated new TSK 0
I20260924 10:07:45.926362 9303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:07:45.987876 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38099
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:46.332410 9319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:46.332702 9319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:46.332878 9319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:46.340381 9319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:46.340504 9319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:46.340698 9319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:07:46.350961 9319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:46.352375 9319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:46.353704 9319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:46.368445 9325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:46.368495 9324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:46.368696 9327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:46.369766 9319 server_base.cc:1034] running on GCE node
I20260924 10:07:46.370685 9319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:46.371840 9319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:46.373211 9319 hybrid_clock.cc:648] HybridClock initialized: now 1790244466373120 us; error 81 us; skew 500 ppm
I20260924 10:07:46.373943 9319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:46 dist-test-slave-s3n3 krb5kdc[9213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244466, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:46.378436 9319 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:07:46.380227 9319 webserver.cc:460] Webserver started at http://127.6.2.193:34443/ using document root <none> and password file <none>
I20260924 10:07:46.380982 9319 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:46.381134 9319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:46.381485 9319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:46.384709 9319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "64de1b88e8654f919255f7cda03866a0"
format_stamp: "Formatted at 2026-09-24 10:07:46 on dist-test-slave-s3n3"
server_key: "4c2b9d4c43d161c01ce79eaa18ae9d0c"
server_key_iv: "1016474381ea5eed27c2895476cb3308"
server_key_version: "encryptionkey@0"
I20260924 10:07:46.385542 9319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "64de1b88e8654f919255f7cda03866a0"
format_stamp: "Formatted at 2026-09-24 10:07:46 on dist-test-slave-s3n3"
server_key: "4c2b9d4c43d161c01ce79eaa18ae9d0c"
server_key_iv: "1016474381ea5eed27c2895476cb3308"
server_key_version: "encryptionkey@0"
I20260924 10:07:46.391031 9319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:07:46.395689 9334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:46.397640 9319 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 10:07:46.397861 9319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "64de1b88e8654f919255f7cda03866a0"
format_stamp: "Formatted at 2026-09-24 10:07:46 on dist-test-slave-s3n3"
server_key: "4c2b9d4c43d161c01ce79eaa18ae9d0c"
server_key_iv: "1016474381ea5eed27c2895476cb3308"
server_key_version: "encryptionkey@0"
I20260924 10:07:46.398085 9319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:46.421310 9319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:46.425539 9319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:46.425858 9319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:46.426853 9319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:46.428220 9319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:46.428325 9319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:46.428484 9319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:46.428556 9319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:46.464386 9319 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:41129
I20260924 10:07:46.464484 9446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:41129 every 8 connection(s)
I20260924 10:07:46.465695 9319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:07:46.468379 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 9319
I20260924 10:07:46.468596 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:07:46.469303 6155 external_mini_cluster.cc:1428] Setting key 6601b76669fb4bea36cdb4803284b726
Sep 24 10:07:46 dist-test-slave-s3n3 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244466, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
I20260924 10:07:46.490375 9449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:46.468429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:34243 (local address 127.6.2.254:38099)
0924 10:07:46.468899 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:46.468907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:46.470320 (+ 1413us) server_negotiation.cc:407] Connection header received
0924 10:07:46.472108 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:46.472113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:46.472197 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:46.472326 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:46.473883 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:46.474600 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:46.476225 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:46.476626 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:46.476767 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:46.482342 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:46.482352 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:46.482361 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:46.482412 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:46.486694 (+ 4282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:46.487712 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:46.487719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:46.487725 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:46.487875 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:46.488405 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:46.488410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:46.488414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:46.488814 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:46.488992 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:46.489837 (+ 845us) server_negotiation.cc:299] Negotiation successful
0924 10:07:46.490036 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":210,"threads_started":1}
I20260924 10:07:46.491468 9450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:46.468999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:38099 (local address 127.6.2.193:34243)
0924 10:07:46.469826 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:46.469881 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:46.471846 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:46.472563 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:46.472571 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:46.473158 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:46.473718 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:46.473735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:46.474774 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:46.474778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:46.476072 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:46.476084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:46.476855 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:46.476864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:46.477199 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:46.477983 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:46.478016 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:46.481910 (+ 3894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:46.486912 (+ 5002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:46.486921 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:46.486940 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:46.487555 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:46.488035 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:46.488039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:46.488042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:46.488258 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:46.488978 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:46.488985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:46.489413 (+ 428us) client_negotiation.cc:769] Sending connection context
0924 10:07:46.489694 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 10:07:46.490120 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":345,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
I20260924 10:07:46.493178 9447 heartbeater.cc:347] Connected to a master server at 127.6.2.254:38099
I20260924 10:07:46.493526 9447 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:46.494400 9447 heartbeater.cc:511] Master 127.6.2.254:38099 requested a full tablet report, sending...
I20260924 10:07:46.496668 9258 ts_manager.cc:194] Registered new tserver with Master: 64de1b88e8654f919255f7cda03866a0 (127.6.2.193:41129)
I20260924 10:07:46.498314 9258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:34243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:07:46.519910 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38099
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:46.848547 9453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:46.848835 9453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:46.848932 9453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:46.856544 9453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:46.856678 9453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:46.856793 9453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:07:46.866895 9453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:46.868253 9453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:46.870198 9453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:46.884848 9458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:46.884948 9459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:46.885038 9461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:46.885848 9453 server_base.cc:1034] running on GCE node
I20260924 10:07:46.886523 9453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:46.887609 9453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:46.888849 9453 hybrid_clock.cc:648] HybridClock initialized: now 1790244466888809 us; error 47 us; skew 500 ppm
I20260924 10:07:46.889222 9453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:46 dist-test-slave-s3n3 krb5kdc[9213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244466, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:46.892824 9453 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:07:46.894747 9453 webserver.cc:460] Webserver started at http://127.6.2.194:34359/ using document root <none> and password file <none>
I20260924 10:07:46.895581 9453 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:46.895687 9453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:46.895932 9453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:46.898934 9453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "a826a1d2956745f89412fb08b5e90551"
format_stamp: "Formatted at 2026-09-24 10:07:46 on dist-test-slave-s3n3"
server_key: "05ca442829023632c79c780ec88b9aa0"
server_key_iv: "9520eef309baa83c6e0ea6bf7307ffd1"
server_key_version: "encryptionkey@0"
I20260924 10:07:46.899825 9453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "a826a1d2956745f89412fb08b5e90551"
format_stamp: "Formatted at 2026-09-24 10:07:46 on dist-test-slave-s3n3"
server_key: "05ca442829023632c79c780ec88b9aa0"
server_key_iv: "9520eef309baa83c6e0ea6bf7307ffd1"
server_key_version: "encryptionkey@0"
I20260924 10:07:46.905395 9453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:07:46.909013 9468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:46.910584 9453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:07:46.910773 9453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "a826a1d2956745f89412fb08b5e90551"
format_stamp: "Formatted at 2026-09-24 10:07:46 on dist-test-slave-s3n3"
server_key: "05ca442829023632c79c780ec88b9aa0"
server_key_iv: "9520eef309baa83c6e0ea6bf7307ffd1"
server_key_version: "encryptionkey@0"
I20260924 10:07:46.910993 9453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:46.929723 9453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:46.934144 9453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:46.934471 9453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:46.935561 9453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:46.936986 9453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:46.937060 9453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:46.937192 9453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:46.937242 9453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:46.973148 9453 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:37875
I20260924 10:07:46.973318 9580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:37875 every 8 connection(s)
I20260924 10:07:46.974483 9453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:07:46.981403 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 9453
I20260924 10:07:46.981617 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:07:46.982077 6155 external_mini_cluster.cc:1428] Setting key 2fe06e0203281c18edb65224e2a1b08a
Sep 24 10:07:46 dist-test-slave-s3n3 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244466, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:46.999760 9449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:46.977130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:35695 (local address 127.6.2.254:38099)
0924 10:07:46.977369 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:46.977375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:46.978529 (+ 1154us) server_negotiation.cc:407] Connection header received
0924 10:07:46.980396 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:46.980402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:46.980477 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:46.980585 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:46.983424 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:46.984456 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:46.986251 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:46.987104 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:46.987260 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:46.992909 (+ 5649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:46.992915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:46.992918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:46.992954 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:46.995732 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:46.997018 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:46.997022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:46.997024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:46.997146 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:46.997714 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:46.997718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:46.997720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:46.998144 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:46.998293 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:46.999315 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0924 10:07:46.999498 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 10:07:47.001459 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:46.977587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:38099 (local address 127.6.2.194:35695)
0924 10:07:46.978355 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:46.978398 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:46.980093 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:46.981279 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:46.981292 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:46.982307 (+ 1015us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:46.983196 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:46.983220 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:46.984675 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:46.984680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:46.986051 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:46.986063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:46.987512 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:46.987520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:46.987852 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:46.988882 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:46.988910 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:46.992447 (+ 3537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:46.995911 (+ 3464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:46.995925 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:46.995949 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:46.996682 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:46.997274 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:46.997282 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:46.997289 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:46.997588 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:46.998305 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:46.998314 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:46.998839 (+ 525us) client_negotiation.cc:769] Sending connection context
0924 10:07:46.999221 (+ 382us) client_negotiation.cc:240] Negotiation successful
0924 10:07:46.999905 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":330,"threads_started":1}
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:07:47.003262 9581 heartbeater.cc:347] Connected to a master server at 127.6.2.254:38099
I20260924 10:07:47.003551 9581 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:47.004331 9581 heartbeater.cc:511] Master 127.6.2.254:38099 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
I20260924 10:07:47.006049 9258 ts_manager.cc:194] Registered new tserver with Master: a826a1d2956745f89412fb08b5e90551 (127.6.2.194:37875)
I20260924 10:07:47.006865 9258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:35695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:07:47.031457 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38099
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:47.391621 9586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:47.392190 9586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:47.392400 9586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:47.401721 9586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:47.401922 9586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:47.402169 9586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:07:47.413993 9586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:47.415522 9586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:47.416859 9586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:47.430585 9594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:47.430781 9591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:47.430727 9592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:47.431828 9586 server_base.cc:1034] running on GCE node
I20260924 10:07:47.432444 9586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:47.433408 9586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:47.434701 9586 hybrid_clock.cc:648] HybridClock initialized: now 1790244467434615 us; error 74 us; skew 500 ppm
I20260924 10:07:47.435325 9586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:47 dist-test-slave-s3n3 krb5kdc[9213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244467, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:47.439532 9586 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:07:47.441256 9586 webserver.cc:460] Webserver started at http://127.6.2.195:36627/ using document root <none> and password file <none>
I20260924 10:07:47.442008 9586 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:47.442157 9586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:47.442471 9586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:47.445533 9586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "119ac65ad112499fa476ac80692bf596"
format_stamp: "Formatted at 2026-09-24 10:07:47 on dist-test-slave-s3n3"
server_key: "f7da099e58bab37cc6e42119d9d38798"
server_key_iv: "88c9acdbc2404d0482541344f5a0efa7"
server_key_version: "encryptionkey@0"
I20260924 10:07:47.446341 9586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "119ac65ad112499fa476ac80692bf596"
format_stamp: "Formatted at 2026-09-24 10:07:47 on dist-test-slave-s3n3"
server_key: "f7da099e58bab37cc6e42119d9d38798"
server_key_iv: "88c9acdbc2404d0482541344f5a0efa7"
server_key_version: "encryptionkey@0"
I20260924 10:07:47.451903 9586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 10:07:47.455600 9601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:47.457099 9586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:07:47.457304 9586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "119ac65ad112499fa476ac80692bf596"
format_stamp: "Formatted at 2026-09-24 10:07:47 on dist-test-slave-s3n3"
server_key: "f7da099e58bab37cc6e42119d9d38798"
server_key_iv: "88c9acdbc2404d0482541344f5a0efa7"
server_key_version: "encryptionkey@0"
I20260924 10:07:47.457582 9586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:47.482501 9586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:47.487382 9586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:47.487705 9586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:47.488690 9586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:47.490063 9586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:47.490140 9586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:47.490269 9586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:47.490319 9586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:47.501406 9447 heartbeater.cc:503] Master 127.6.2.254:38099 was elected leader, sending a full tablet report...
I20260924 10:07:47.525902 9586 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:38059
I20260924 10:07:47.526000 9713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:38059 every 8 connection(s)
I20260924 10:07:47.527308 9586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:07:47.533381 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 9586
I20260924 10:07:47.533609 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:07:47.534210 6155 external_mini_cluster.cc:1428] Setting key ddf023b472909956ecce0b33f3f9adb2
Sep 24 10:07:47 dist-test-slave-s3n3 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244467, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:47.552345 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:47.531282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:35107 (local address 127.6.2.254:38099)
0924 10:07:47.531814 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:47.531819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:47.533400 (+ 1581us) server_negotiation.cc:407] Connection header received
0924 10:07:47.535886 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:47.535893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:47.535987 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:47.536106 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:47.538214 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:47.539009 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:47.540627 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:47.541027 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:47.541132 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:47.544785 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:47.544793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:47.544797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:47.544850 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:47.547868 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:47.549393 (+ 1525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:47.549397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:47.549400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:47.549597 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:47.550229 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:47.550238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:47.550243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:47.550633 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:47.550822 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:47.551675 (+ 853us) server_negotiation.cc:299] Negotiation successful
0924 10:07:47.551924 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":319,"threads_started":1}
I20260924 10:07:47.553457 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:47.532039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:38099 (local address 127.6.2.195:35107)
0924 10:07:47.533049 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:47.533112 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:47.535539 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:47.536475 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:47.536488 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:47.537212 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:47.538038 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:47.538069 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:47.539202 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:47.539206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:47.540475 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:47.540487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:47.541244 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:47.541249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:47.541480 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:47.542198 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:47.542222 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:47.544385 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:47.548175 (+ 3790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:47.548189 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:47.548227 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:47.549086 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:47.549791 (+ 705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:47.549801 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:47.549806 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:47.550087 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:47.550798 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:47.550808 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:47.551271 (+ 463us) client_negotiation.cc:769] Sending connection context
0924 10:07:47.551584 (+ 313us) client_negotiation.cc:240] Negotiation successful
0924 10:07:47.552194 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":403,"threads_started":1}
I20260924 10:07:47.554785 9714 heartbeater.cc:347] Connected to a master server at 127.6.2.254:38099
I20260924 10:07:47.555151 9714 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:47.555953 9714 heartbeater.cc:511] Master 127.6.2.254:38099 requested a full tablet report, sending...
I20260924 10:07:47.557657 9258 ts_manager.cc:194] Registered new tserver with Master: 119ac65ad112499fa476ac80692bf596 (127.6.2.195:38059)
I20260924 10:07:47.558794 9258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:35107
I20260924 10:07:47.564206 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:07:47 dist-test-slave-s3n3 krb5kdc[9213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244467, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 10:07:47 dist-test-slave-s3n3 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244467, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:47.619930 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:47.602273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.6.2.254:38099)
0924 10:07:47.602790 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:47.602801 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:47.603358 (+ 557us) server_negotiation.cc:407] Connection header received
0924 10:07:47.603690 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:47.603696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:47.603778 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:47.603892 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:47.605488 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:47.606835 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:47.608398 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:47.608848 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:47.608966 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:47.613557 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:47.613563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:47.613566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:47.613613 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:47.617056 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:47.617881 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:47.617885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:47.617888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:47.618022 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:47.618412 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:47.618415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:47.618417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:47.618755 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:47.618940 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:47.619342 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 10:07:47.619580 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
W20260924 10:07:47.620498 9258 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47968
I20260924 10:07:47.622584 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 9319
Sep 24 10:07:47 dist-test-slave-s3n3 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244467, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:07:47.640419 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:47.623956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.6.2.193:41129)
0924 10:07:47.625335 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:47.625342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:47.625374 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:07:47.625435 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:47.625441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:47.625643 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:47.625783 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:47.627167 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:47.628215 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:47.629508 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:47.630133 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:47.630246 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:47.634517 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:47.634529 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:47.634545 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:47.634582 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:47.637599 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:47.638351 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:47.638358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:47.638360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:47.638501 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:47.638854 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:47.638858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:47.638860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:47.639315 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:47.639546 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:47.639925 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 10:07:47.640133 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1221,"thread_start_us":1065,"threads_started":1}
W20260924 10:07:47.641788 9442 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58082
W20260924 10:07:47.642992 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 9319: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:07:47.643285 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 9319
I20260924 10:07:47.659606 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 9453
Sep 24 10:07:47 dist-test-slave-s3n3 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244467, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:07:47.680073 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:47.660550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39394 (local address 127.6.2.194:37875)
0924 10:07:47.661048 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:47.661055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:47.661081 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:07:47.661148 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:47.661154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:47.661441 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:47.661640 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:47.662944 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:47.664768 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:47.666187 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:47.666793 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:47.666963 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:47.673414 (+ 6451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:47.673438 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:47.673461 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:47.673514 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:47.676984 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:47.677837 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:47.677851 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:47.677854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:47.677985 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:47.678365 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:47.678372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:47.678375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:47.678735 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:47.679064 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:47.679402 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 10:07:47.679685 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":243,"threads_started":1}
W20260924 10:07:47.681471 9576 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39394
W20260924 10:07:47.682682 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 9453: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:07:47.682873 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 9453
I20260924 10:07:47.698745 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 9586
Sep 24 10:07:47 dist-test-slave-s3n3 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244467, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:07:47.719357 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:47.700196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.6.2.195:38059)
0924 10:07:47.701317 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:47.701326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:47.701369 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:07:47.701461 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:47.701468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:47.701757 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:47.701962 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:47.703984 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:47.705097 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:47.706514 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:47.707207 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:47.707332 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:47.712831 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:47.712882 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:47.712900 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:47.712948 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:47.716337 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:47.717212 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:47.717220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:47.717222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:47.717381 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:47.717730 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:47.717733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:47.717735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:47.718110 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:47.718345 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:47.718682 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 10:07:47.718893 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":713,"threads_started":1}
W20260924 10:07:47.720739 9709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46614
W20260924 10:07:47.721912 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 9586: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:07:47.722087 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 9586
I20260924 10:07:47.737994 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 9226
I20260924 10:07:47.739524 9288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:47.850286 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 9226
2026-09-24T10:07:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3775 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:07:47 dist-test-slave-s3n3 krb5kdc[9732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:07:47 dist-test-slave-s3n3 krb5kdc[9732](info): set up 2 sockets
Sep 24 10:07:47 dist-test-slave-s3n3 krb5kdc[9732](info): commencing operation
krb5kdc: starting...
W20260924 10:07:48.979807 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:07:49 dist-test-slave-s3n3 krb5kdc[9732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:07:49.200212 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:39435
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:39435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:49.537849 9745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:49.538301 9745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:49.538471 9745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:49.545981 9745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:49.546116 9745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:49.546196 9745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:49.546277 9745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:49.546358 9745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:49.556799 9745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:39435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:39435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:49.558239 9745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:49.559640 9745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:49.573110 9750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:49.573177 9753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:49.573153 9751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:49.574009 9745 server_base.cc:1034] running on GCE node
I20260924 10:07:49.574862 9745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:49.576450 9745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:49.577696 9745 hybrid_clock.cc:648] HybridClock initialized: now 1790244469577630 us; error 68 us; skew 500 ppm
I20260924 10:07:49.578088 9745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:49 dist-test-slave-s3n3 krb5kdc[9732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244469, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:49.589082 9745 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:07:49.591737 9745 webserver.cc:460] Webserver started at http://127.6.2.254:37545/ using document root <none> and password file <none>
I20260924 10:07:49.592571 9745 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:49.592707 9745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:49.593039 9745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:49.596230 9745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "9d5d8be6983f4c6da2a83d24c7865cd0"
format_stamp: "Formatted at 2026-09-24 10:07:49 on dist-test-slave-s3n3"
server_key: "e58ff76efea5c35882e7f6683dadb650"
server_key_iv: "5af1cca3b6d882dddc81a5e0d9bbdb39"
server_key_version: "encryptionkey@0"
I20260924 10:07:49.597048 9745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "9d5d8be6983f4c6da2a83d24c7865cd0"
format_stamp: "Formatted at 2026-09-24 10:07:49 on dist-test-slave-s3n3"
server_key: "e58ff76efea5c35882e7f6683dadb650"
server_key_iv: "5af1cca3b6d882dddc81a5e0d9bbdb39"
server_key_version: "encryptionkey@0"
I20260924 10:07:49.602725 9745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:07:49.606626 9760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:49.609002 9745 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:07:49.609251 9745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "9d5d8be6983f4c6da2a83d24c7865cd0"
format_stamp: "Formatted at 2026-09-24 10:07:49 on dist-test-slave-s3n3"
server_key: "e58ff76efea5c35882e7f6683dadb650"
server_key_iv: "5af1cca3b6d882dddc81a5e0d9bbdb39"
server_key_version: "encryptionkey@0"
I20260924 10:07:49.609494 9745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:49.628326 9745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:49.632856 9745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:49.633168 9745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:49.653712 9745 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:39435
I20260924 10:07:49.653756 9811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:39435 every 8 connection(s)
I20260924 10:07:49.655901 9745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:49.660190 9812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:49.661993 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 9745
I20260924 10:07:49.662251 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:07:49.663101 6155 external_mini_cluster.cc:1428] Setting key cfa5dd44d48fe972a8cddc4217879c7a
I20260924 10:07:49.673791 9812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0: Bootstrap starting.
I20260924 10:07:49.677978 9812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:49.679409 9812 log.cc:824] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:07:49 dist-test-slave-s3n3 krb5kdc[9732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:49.682725 9812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0: No bootstrap required, opened a new log
I20260924 10:07:49.688372 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:49.665381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49304 (local address 127.6.2.254:39435)
0924 10:07:49.666321 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:49.666334 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:49.666384 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:07:49.668203 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:49.668227 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:49.668668 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:49.668998 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:49.671035 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:49.672884 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:49.674807 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:49.675672 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:49.675827 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:49.680790 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:49.680799 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:49.680816 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:49.680853 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:49.684171 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:49.684927 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:49.684945 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:49.684960 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:49.685132 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:49.685526 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:49.685532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:49.685535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:49.686092 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:49.686381 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:49.686856 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 10:07:49.687316 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":479,"threads_started":1}
I20260924 10:07:49.691679 9812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5d8be6983f4c6da2a83d24c7865cd0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39435 } }
I20260924 10:07:49.692086 9812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:49.692211 9812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5d8be6983f4c6da2a83d24c7865cd0, State: Initialized, Role: FOLLOWER
I20260924 10:07:49.692873 9812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5d8be6983f4c6da2a83d24c7865cd0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39435 } }
I20260924 10:07:49.693116 9812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:49.693250 9812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:49.693418 9812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:49.696139 9812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5d8be6983f4c6da2a83d24c7865cd0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39435 } }
I20260924 10:07:49.696653 9812 leader_election.cc:302] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d5d8be6983f4c6da2a83d24c7865cd0; no voters:
I20260924 10:07:49.697178 9812 leader_election.cc:288] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:49.697324 9817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:49.699203 9817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [term 1 LEADER]: Becoming Leader. State: Replica: 9d5d8be6983f4c6da2a83d24c7865cd0, State: Running, Role: LEADER
I20260924 10:07:49.699407 9812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:49.699751 9817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5d8be6983f4c6da2a83d24c7865cd0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39435 } }
I20260924 10:07:49.703349 9819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d5d8be6983f4c6da2a83d24c7865cd0. Latest consensus state: current_term: 1 leader_uuid: "9d5d8be6983f4c6da2a83d24c7865cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5d8be6983f4c6da2a83d24c7865cd0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39435 } } }
I20260924 10:07:49.704145 9819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:49.704047 9818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d5d8be6983f4c6da2a83d24c7865cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5d8be6983f4c6da2a83d24c7865cd0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39435 } } }
I20260924 10:07:49.704538 9818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:49.707690 9827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:49.711930 9827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:49.720810 9827 catalog_manager.cc:1347] Generated new cluster ID: f1bb738ff9ea43be86c24aaee0d916ca
I20260924 10:07:49.720952 9827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:49.747663 9827 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:49.749056 9827 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:49.765393 9827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d5d8be6983f4c6da2a83d24c7865cd0: Generated new TSK 0
I20260924 10:07:49.766507 9827 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:07:49.820794 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:39435
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:50.158403 9838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:50.158725 9838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:50.158819 9838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:50.169243 9838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:50.169497 9838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:50.169752 9838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:07:50.185037 9838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:39435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:50.186739 9838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:50.188546 9838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:50.204389 9843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:50.204980 9846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:50.205354 9844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:50.206676 9838 server_base.cc:1034] running on GCE node
I20260924 10:07:50.207525 9838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:50.208626 9838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:50.209839 9838 hybrid_clock.cc:648] HybridClock initialized: now 1790244470209773 us; error 59 us; skew 500 ppm
I20260924 10:07:50.210227 9838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:50 dist-test-slave-s3n3 krb5kdc[9732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244470, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:50.214215 9838 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:07:50.216190 9838 webserver.cc:460] Webserver started at http://127.6.2.193:35773/ using document root <none> and password file <none>
I20260924 10:07:50.216951 9838 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:50.217085 9838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:50.217435 9838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:50.220618 9838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "b6a2f93ee0854d8bb75fe58db5e7838e"
format_stamp: "Formatted at 2026-09-24 10:07:50 on dist-test-slave-s3n3"
server_key: "d3abb8d7fddfe8372ff93836a4ca8875"
server_key_iv: "d4e277c2e3f3c55e08f1b5e964ef308d"
server_key_version: "encryptionkey@0"
I20260924 10:07:50.221443 9838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "b6a2f93ee0854d8bb75fe58db5e7838e"
format_stamp: "Formatted at 2026-09-24 10:07:50 on dist-test-slave-s3n3"
server_key: "d3abb8d7fddfe8372ff93836a4ca8875"
server_key_iv: "d4e277c2e3f3c55e08f1b5e964ef308d"
server_key_version: "encryptionkey@0"
I20260924 10:07:50.227429 9838 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260924 10:07:50.231637 9853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:50.233346 9838 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:07:50.233546 9838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "b6a2f93ee0854d8bb75fe58db5e7838e"
format_stamp: "Formatted at 2026-09-24 10:07:50 on dist-test-slave-s3n3"
server_key: "d3abb8d7fddfe8372ff93836a4ca8875"
server_key_iv: "d4e277c2e3f3c55e08f1b5e964ef308d"
server_key_version: "encryptionkey@0"
I20260924 10:07:50.233767 9838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:50.257067 9838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:50.261705 9838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:50.262014 9838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:50.263047 9838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:50.264716 9838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:50.264827 9838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:50.264977 9838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:50.265056 9838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:50.301455 9838 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:38549
I20260924 10:07:50.301548 9965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:38549 every 8 connection(s)
I20260924 10:07:50.302889 9838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:07:50.308351 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 9838
I20260924 10:07:50.308604 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:07:50.309250 6155 external_mini_cluster.cc:1428] Setting key f98192fdd7f5c21d05d3121c8ee0a25f
Sep 24 10:07:50 dist-test-slave-s3n3 krb5kdc[9732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244470, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:50.324934 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:50.305605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:33181 (local address 127.6.2.254:39435)
0924 10:07:50.306041 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:50.306047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:50.307123 (+ 1076us) server_negotiation.cc:407] Connection header received
0924 10:07:50.309128 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:50.309134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:50.309237 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:50.309360 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:50.311026 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:50.312019 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:50.313576 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:50.313989 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:50.314130 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:50.317844 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:50.317850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:50.317853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:50.317887 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:50.321436 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:50.322241 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:50.322249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:50.322255 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:50.322400 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:50.322823 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:50.322830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:50.322836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:50.323246 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:50.323401 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:50.324182 (+ 781us) server_negotiation.cc:299] Negotiation successful
0924 10:07:50.324435 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":193,"threads_started":1}
I20260924 10:07:50.325649 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:50.306194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:39435 (local address 127.6.2.193:33181)
0924 10:07:50.306933 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:50.306979 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:50.308849 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:50.309602 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:50.309610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:50.310263 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:50.310828 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:50.310849 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:50.312181 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:50.312187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:50.313410 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:50.313423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:50.314216 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:50.314224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:50.314478 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:50.315300 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:50.315322 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:50.317605 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:50.321616 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:50.321623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:50.321638 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:50.322096 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:50.322524 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:50.322527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:50.322529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:50.322703 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:50.323397 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:50.323403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:50.323767 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 10:07:50.324031 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 10:07:50.324474 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":311,"threads_started":1}
I20260924 10:07:50.327381 9966 heartbeater.cc:347] Connected to a master server at 127.6.2.254:39435
I20260924 10:07:50.327754 9966 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:50.328547 9966 heartbeater.cc:511] Master 127.6.2.254:39435 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
I20260924 10:07:50.331765 9777 ts_manager.cc:194] Registered new tserver with Master: b6a2f93ee0854d8bb75fe58db5e7838e (127.6.2.193:38549)
I20260924 10:07:50.333531 9777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:33181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:07:50.360244 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:39435
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:50.698120 9972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:50.698796 9972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:50.698931 9972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:50.707815 9972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:50.707962 9972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:50.708148 9972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:07:50.719862 9972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:39435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:50.721215 9972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:50.722503 9972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:50.736160 9977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:50.736130 9980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:50.736227 9978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:50.741269 9972 server_base.cc:1034] running on GCE node
I20260924 10:07:50.742018 9972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:50.743167 9972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:50.744438 9972 hybrid_clock.cc:648] HybridClock initialized: now 1790244470744373 us; error 70 us; skew 500 ppm
I20260924 10:07:50.744903 9972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:50 dist-test-slave-s3n3 krb5kdc[9732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244470, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:50.748368 9972 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:07:50.750006 9972 webserver.cc:460] Webserver started at http://127.6.2.194:39425/ using document root <none> and password file <none>
I20260924 10:07:50.750900 9972 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:50.751130 9972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:50.751494 9972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:50.754593 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "e88433b596dd46d1970d8607929bcdac"
format_stamp: "Formatted at 2026-09-24 10:07:50 on dist-test-slave-s3n3"
server_key: "190342b23717800f39ea147486790a67"
server_key_iv: "d164f0ed5c9569b0047bc10c6f93cd8f"
server_key_version: "encryptionkey@0"
I20260924 10:07:50.755446 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "e88433b596dd46d1970d8607929bcdac"
format_stamp: "Formatted at 2026-09-24 10:07:50 on dist-test-slave-s3n3"
server_key: "190342b23717800f39ea147486790a67"
server_key_iv: "d164f0ed5c9569b0047bc10c6f93cd8f"
server_key_version: "encryptionkey@0"
I20260924 10:07:50.760839 9972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260924 10:07:50.764477 9987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:50.766081 9972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:50.766271 9972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "e88433b596dd46d1970d8607929bcdac"
format_stamp: "Formatted at 2026-09-24 10:07:50 on dist-test-slave-s3n3"
server_key: "190342b23717800f39ea147486790a67"
server_key_iv: "d164f0ed5c9569b0047bc10c6f93cd8f"
server_key_version: "encryptionkey@0"
I20260924 10:07:50.766496 9972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:50.798806 9972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:50.804738 9972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:50.805059 9972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:50.806118 9972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:50.807890 9972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:50.808001 9972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:50.808152 9972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:50.808233 9972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:50.845202 9972 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:39811
I20260924 10:07:50.845252 10099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:39811 every 8 connection(s)
I20260924 10:07:50.846539 9972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:07:50.851831 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 9972
I20260924 10:07:50.852110 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:07:50.853004 6155 external_mini_cluster.cc:1428] Setting key 332968981d3daa2513c03e5eac53204d
Sep 24 10:07:50 dist-test-slave-s3n3 krb5kdc[9732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244470, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:07:50.874282 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:50.849970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:52437 (local address 127.6.2.254:39435)
0924 10:07:50.850412 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:50.850416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:50.852044 (+ 1628us) server_negotiation.cc:407] Connection header received
0924 10:07:50.855715 (+ 3671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:50.855724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:50.855834 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:50.855919 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:50.857720 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:50.858873 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:50.860683 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:50.861504 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:50.861608 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:50.867803 (+ 6195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:50.867809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:50.867812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:50.867853 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:50.870957 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:50.871846 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:50.871852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:50.871856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:50.871999 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:50.872454 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:50.872465 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:50.872467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:50.872738 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:50.872898 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:50.873655 (+ 757us) server_negotiation.cc:299] Negotiation successful
0924 10:07:50.873880 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":241,"threads_started":1}
I20260924 10:07:50.875478 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:50.850515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:39435 (local address 127.6.2.194:52437)
0924 10:07:50.851760 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:50.851849 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:50.855125 (+ 3276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:50.856275 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:50.856286 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:50.856886 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:50.857521 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:50.857546 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:50.859152 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:50.859160 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:50.860527 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:50.860541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:50.861927 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:50.861940 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:50.862484 (+ 544us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:50.863771 (+ 1287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:50.863818 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:50.867465 (+ 3647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:50.871175 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:50.871204 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:50.871229 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:50.871705 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:50.872121 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:50.872126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:50.872130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:50.872329 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:50.872870 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:50.872880 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:50.873267 (+ 387us) client_negotiation.cc:769] Sending connection context
0924 10:07:50.873565 (+ 298us) client_negotiation.cc:240] Negotiation successful
0924 10:07:50.874139 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":366,"threads_started":1}
I20260924 10:07:50.877422 10100 heartbeater.cc:347] Connected to a master server at 127.6.2.254:39435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
I20260924 10:07:50.877871 10100 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:50.878832 10100 heartbeater.cc:511] Master 127.6.2.254:39435 requested a full tablet report, sending...
I20260924 10:07:50.880661 9777 ts_manager.cc:194] Registered new tserver with Master: e88433b596dd46d1970d8607929bcdac (127.6.2.194:39811)
I20260924 10:07:50.881522 9777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:52437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:07:50.908128 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:39435
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:51.254372 10106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:51.254685 10106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:51.254779 10106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:51.262275 10106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:51.262378 10106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:51.262490 10106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:07:51.273183 10106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:39435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:51.274490 10106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:51.275893 10106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:51.290810 10111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:51.290824 10114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:51.291222 10112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:51.292671 10106 server_base.cc:1034] running on GCE node
I20260924 10:07:51.293381 10106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:51.294468 10106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:51.295718 10106 hybrid_clock.cc:648] HybridClock initialized: now 1790244471295669 us; error 73 us; skew 500 ppm
I20260924 10:07:51.296110 10106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:51 dist-test-slave-s3n3 krb5kdc[9732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244471, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:51.299662 10106 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:07:51.301381 10106 webserver.cc:460] Webserver started at http://127.6.2.195:34121/ using document root <none> and password file <none>
I20260924 10:07:51.302124 10106 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:51.302284 10106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:51.302601 10106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:51.305833 10106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "99589c5d41424a4cb71d6bf57c6a0944"
format_stamp: "Formatted at 2026-09-24 10:07:51 on dist-test-slave-s3n3"
server_key: "91087ee324c6d3694dc41f31d51379ed"
server_key_iv: "6ef78ac582396374752aeb664e8a8c7a"
server_key_version: "encryptionkey@0"
I20260924 10:07:51.306651 10106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "99589c5d41424a4cb71d6bf57c6a0944"
format_stamp: "Formatted at 2026-09-24 10:07:51 on dist-test-slave-s3n3"
server_key: "91087ee324c6d3694dc41f31d51379ed"
server_key_iv: "6ef78ac582396374752aeb664e8a8c7a"
server_key_version: "encryptionkey@0"
I20260924 10:07:51.312021 10106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 10:07:51.315757 10121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:51.317132 10106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:51.317317 10106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "99589c5d41424a4cb71d6bf57c6a0944"
format_stamp: "Formatted at 2026-09-24 10:07:51 on dist-test-slave-s3n3"
server_key: "91087ee324c6d3694dc41f31d51379ed"
server_key_iv: "6ef78ac582396374752aeb664e8a8c7a"
server_key_version: "encryptionkey@0"
I20260924 10:07:51.317574 10106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:51.338380 9966 heartbeater.cc:503] Master 127.6.2.254:39435 was elected leader, sending a full tablet report...
I20260924 10:07:51.341552 10106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:51.345896 10106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:51.346143 10106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:51.347194 10106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:51.348489 10106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:51.348559 10106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:51.348650 10106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:51.348693 10106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:51.386983 10106 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:35495
I20260924 10:07:51.387077 10233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:35495 every 8 connection(s)
I20260924 10:07:51.388332 10106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:07:51.389566 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 10106
I20260924 10:07:51.389775 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:07:51.390367 6155 external_mini_cluster.cc:1428] Setting key bb2254c90eecf94367ee351bff3953c7
Sep 24 10:07:51 dist-test-slave-s3n3 krb5kdc[9732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244471, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:51.410439 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:51.391109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:36963 (local address 127.6.2.254:39435)
0924 10:07:51.391625 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:51.391631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:51.392715 (+ 1084us) server_negotiation.cc:407] Connection header received
0924 10:07:51.394805 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:51.394810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:51.394898 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:51.395023 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:51.396563 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.397338 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.398948 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.399452 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.399628 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:51.403474 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:51.403480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:51.403483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:51.403516 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:51.406634 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:51.407539 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:51.407543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:51.407546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:51.407660 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:51.408085 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:51.408090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:51.408092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:51.408380 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:51.408502 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:51.409979 (+ 1477us) server_negotiation.cc:299] Negotiation successful
0924 10:07:51.410148 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":255,"threads_started":1}
I20260924 10:07:51.411656 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:51.391713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:39435 (local address 127.6.2.195:36963)
0924 10:07:51.392562 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:51.392606 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:51.394510 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:51.395290 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:51.395299 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:51.395831 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:51.396388 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:51.396404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.397519 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.397523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:51.398768 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:51.398781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.399722 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.399727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:51.399980 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:51.400717 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:51.400740 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:51.403216 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:51.406783 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:51.406791 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:51.406808 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:51.407352 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:51.407761 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:51.407764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:51.407767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:51.407951 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:51.408609 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:51.408619 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:51.409407 (+ 788us) client_negotiation.cc:769] Sending connection context
0924 10:07:51.409898 (+ 491us) client_negotiation.cc:240] Negotiation successful
0924 10:07:51.410371 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":315,"threads_started":1}
I20260924 10:07:51.413542 10234 heartbeater.cc:347] Connected to a master server at 127.6.2.254:39435
I20260924 10:07:51.413877 10234 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:51.414674 10234 heartbeater.cc:511] Master 127.6.2.254:39435 requested a full tablet report, sending...
I20260924 10:07:51.416644 9777 ts_manager.cc:194] Registered new tserver with Master: 99589c5d41424a4cb71d6bf57c6a0944 (127.6.2.195:35495)
I20260924 10:07:51.417968 9777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:36963
I20260924 10:07:51.418855 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:07:51 dist-test-slave-s3n3 krb5kdc[9732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:07:51 dist-test-slave-s3n3 krb5kdc[9732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:07:51.473249 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:51.456477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.6.2.193:38549)
0924 10:07:51.457088 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:51.457095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:51.457122 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:07:51.457189 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:51.457196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:51.457432 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:51.457587 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:51.458954 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.460082 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.461353 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.461930 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.462039 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:51.466909 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:51.466924 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:51.466941 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:51.466980 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:51.470327 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:51.471349 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:51.471357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:51.471360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:51.471501 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:51.471862 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:51.471865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:51.471867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:51.472236 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:51.472489 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:51.472760 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 10:07:51.472965 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":334,"threads_started":1}
Sep 24 10:07:51 dist-test-slave-s3n3 krb5kdc[9732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:51.494683 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:51.478309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.6.2.254:39435)
0924 10:07:51.478695 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:51.478704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:51.479198 (+ 494us) server_negotiation.cc:407] Connection header received
0924 10:07:51.479383 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:51.479390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:51.479466 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:51.479578 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:51.481057 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.482135 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.483865 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.484324 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.484508 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:51.489127 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:51.489134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:51.489138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:51.489182 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:51.491986 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:51.492721 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:51.492733 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:51.492735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:51.492849 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:51.493229 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:51.493234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:51.493237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:51.493576 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:51.493833 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:51.494128 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 10:07:51.494364 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
W20260924 10:07:51.495539 9777 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49310
I20260924 10:07:51.496740 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 9838
I20260924 10:07:51.510862 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:51.497704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.6.2.193:38549)
0924 10:07:51.497869 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:51.497873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:51.498605 (+ 732us) server_negotiation.cc:407] Connection header received
0924 10:07:51.498822 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:51.498826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:51.498918 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:51.499026 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:51.500381 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.501185 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.502381 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.502874 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.503011 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:51.505416 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:51.505422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:51.505427 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:51.505467 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:51.508305 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:51.508972 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:51.508979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:51.508982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:51.509112 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:51.509471 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:51.509477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:51.509480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:51.509738 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:51.509983 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:51.510303 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 10:07:51.510548 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:07:51.511698 9961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:51.598456 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 9838
I20260924 10:07:51.616114 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 9972
Sep 24 10:07:51 dist-test-slave-s3n3 krb5kdc[9732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:07:51.636533 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:51.617114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.6.2.194:39811)
0924 10:07:51.617715 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:51.617723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:51.617749 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:07:51.617814 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:51.617820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:51.618342 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:51.618625 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:51.619992 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.621159 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.622798 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.623507 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.623633 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:51.629041 (+ 5408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:51.629062 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:51.629084 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:51.629138 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:51.632750 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:51.633573 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:51.633581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:51.633584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:51.633738 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:51.634170 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:51.634174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:51.634176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:51.634558 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:51.634803 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:51.635801 (+ 998us) server_negotiation.cc:299] Negotiation successful
0924 10:07:51.636170 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":317,"threads_started":1}
I20260924 10:07:51.637893 10095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:51.730253 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 9972
I20260924 10:07:51.749790 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 10106
Sep 24 10:07:51 dist-test-slave-s3n3 krb5kdc[9732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:07:51.769428 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:51.751428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.6.2.195:35495)
0924 10:07:51.752035 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:51.752043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:51.752071 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:07:51.752131 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:51.752138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:51.752357 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:51.752560 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:51.753818 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.754840 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.756230 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:51.756814 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:51.756956 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:51.761825 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:51.761838 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:51.761856 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:51.761895 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:51.766217 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:51.767046 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:51.767053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:51.767056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:51.767255 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:51.767712 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:51.767717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:51.767720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:51.768101 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:51.768457 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:51.768769 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 10:07:51.769056 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":312,"threads_started":1}
I20260924 10:07:51.770781 10229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:51.858999 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 10106
I20260924 10:07:51.876581 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 9745
I20260924 10:07:51.878201 9807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:51.984540 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 9745
2026-09-24T10:07:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4131 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:07:52 dist-test-slave-s3n3 krb5kdc[10254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:07:52 dist-test-slave-s3n3 krb5kdc[10254](info): set up 2 sockets
Sep 24 10:07:52 dist-test-slave-s3n3 krb5kdc[10254](info): commencing operation
krb5kdc: starting...
W20260924 10:07:53.091604 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:07:53 dist-test-slave-s3n3 krb5kdc[10254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:07:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:07:53.300125 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:45279
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:45279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:53.628753 10267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:53.629076 10267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:53.629173 10267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:53.636422 10267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:53.636545 10267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:53.636591 10267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:53.636646 10267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:53.636696 10267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:53.646667 10267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:45279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:45279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:53.647970 10267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:53.649246 10267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:53.661015 10272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:53.660980 10273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:53.660913 10275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:53.662253 10267 server_base.cc:1034] running on GCE node
I20260924 10:07:53.663206 10267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:53.664814 10267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:53.666095 10267 hybrid_clock.cc:648] HybridClock initialized: now 1790244473666040 us; error 79 us; skew 500 ppm
I20260924 10:07:53.666491 10267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:53 dist-test-slave-s3n3 krb5kdc[10254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244473, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:53.669798 10267 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:07:53.671103 10267 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:53.671290 10267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:53.671636 10267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:53.674659 10267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "ae42931870ab4ff6ab6ee41f79255621"
format_stamp: "Formatted at 2026-09-24 10:07:53 on dist-test-slave-s3n3"
server_key: "74ed17c66221f80ab2d8d83be4bed899"
server_key_iv: "df35c206a5477e93e75227fa24ed50e2"
server_key_version: "encryptionkey@0"
I20260924 10:07:53.675566 10267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "ae42931870ab4ff6ab6ee41f79255621"
format_stamp: "Formatted at 2026-09-24 10:07:53 on dist-test-slave-s3n3"
server_key: "74ed17c66221f80ab2d8d83be4bed899"
server_key_iv: "df35c206a5477e93e75227fa24ed50e2"
server_key_version: "encryptionkey@0"
I20260924 10:07:53.680925 10267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 10:07:53.684711 10281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:53.687018 10267 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:07:53.687269 10267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "ae42931870ab4ff6ab6ee41f79255621"
format_stamp: "Formatted at 2026-09-24 10:07:53 on dist-test-slave-s3n3"
server_key: "74ed17c66221f80ab2d8d83be4bed899"
server_key_iv: "df35c206a5477e93e75227fa24ed50e2"
server_key_version: "encryptionkey@0"
I20260924 10:07:53.687507 10267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:53.717885 10267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:53.723788 10267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:53.724081 10267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:53.744719 10267 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:45279
I20260924 10:07:53.744742 10332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:45279 every 8 connection(s)
I20260924 10:07:53.745684 10267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:53.749475 10333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:53.752940 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 10267
I20260924 10:07:53.753204 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:07:53.753968 6155 external_mini_cluster.cc:1428] Setting key 5ec73dec480bd22098f2f211ce94f2b3
I20260924 10:07:53.763254 10333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621: Bootstrap starting.
I20260924 10:07:53.767190 10333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:53.768610 10333 log.cc:824] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:07:53 dist-test-slave-s3n3 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:53.771770 10333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621: No bootstrap required, opened a new log
I20260924 10:07:53.779354 10336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:53.756257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.6.2.254:45279)
0924 10:07:53.757058 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:53.757070 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:53.757112 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:07:53.758791 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:53.758815 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:53.759305 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:53.759644 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:53.761599 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:53.763798 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:53.765827 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:53.766739 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:53.767014 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:53.771722 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:53.771731 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:53.771747 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:53.771792 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:53.775042 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:53.775804 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:53.775816 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:53.775831 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:53.776004 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:53.776392 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:53.776397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:53.776401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:53.777002 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:53.777326 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:53.777804 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 10:07:53.778244 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":391,"threads_started":1}
I20260924 10:07:53.779853 10333 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae42931870ab4ff6ab6ee41f79255621" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 45279 } }
I20260924 10:07:53.780294 10333 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:53.780412 10333 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae42931870ab4ff6ab6ee41f79255621, State: Initialized, Role: FOLLOWER
I20260924 10:07:53.781051 10333 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae42931870ab4ff6ab6ee41f79255621" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 45279 } }
I20260924 10:07:53.781288 10333 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:53.781445 10333 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:53.781641 10333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:53.784495 10333 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae42931870ab4ff6ab6ee41f79255621" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 45279 } }
I20260924 10:07:53.785069 10333 leader_election.cc:302] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae42931870ab4ff6ab6ee41f79255621; no voters:
I20260924 10:07:53.785645 10333 leader_election.cc:288] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:53.785782 10338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:53.786558 10338 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [term 1 LEADER]: Becoming Leader. State: Replica: ae42931870ab4ff6ab6ee41f79255621, State: Running, Role: LEADER
I20260924 10:07:53.787155 10338 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae42931870ab4ff6ab6ee41f79255621" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 45279 } }
I20260924 10:07:53.787860 10333 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:53.790050 10339 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae42931870ab4ff6ab6ee41f79255621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae42931870ab4ff6ab6ee41f79255621" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 45279 } } }
I20260924 10:07:53.790377 10339 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:53.791330 10345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:53.791154 10340 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae42931870ab4ff6ab6ee41f79255621. Latest consensus state: current_term: 1 leader_uuid: "ae42931870ab4ff6ab6ee41f79255621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae42931870ab4ff6ab6ee41f79255621" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 45279 } } }
I20260924 10:07:53.791575 10340 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:53.798141 10345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:53.807040 10345 catalog_manager.cc:1347] Generated new cluster ID: dcbba78ee9d74a889544d1289f1d17ed
I20260924 10:07:53.807183 10345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:53.821142 10345 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:53.823045 10345 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:53.830734 10345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621: Generated new TSK 0
I20260924 10:07:53.831652 10345 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:07:53.891867 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:45279
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:54.251602 10359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:54.251917 10359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:54.252019 10359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:54.260224 10359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:54.260352 10359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:54.260475 10359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:07:54.270603 10359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:45279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:54.272003 10359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:54.273275 10359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:54.287959 10364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:54.288004 10367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:54.288197 10365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:54.288640 10359 server_base.cc:1034] running on GCE node
I20260924 10:07:54.289451 10359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:54.290468 10359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:54.291683 10359 hybrid_clock.cc:648] HybridClock initialized: now 1790244474291642 us; error 38 us; skew 500 ppm
I20260924 10:07:54.292024 10359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:54 dist-test-slave-s3n3 krb5kdc[10254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:54.295495 10359 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:07:54.296706 10359 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:54.296837 10359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:54.297165 10359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:54.300246 10359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "48ff35ff14744390aa0544449c34ff04"
format_stamp: "Formatted at 2026-09-24 10:07:54 on dist-test-slave-s3n3"
server_key: "47cf83ddae9b64fdd0d541635a58c882"
server_key_iv: "1ca0b04d9a42b4a3dd8c7937cb462586"
server_key_version: "encryptionkey@0"
I20260924 10:07:54.301062 10359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "48ff35ff14744390aa0544449c34ff04"
format_stamp: "Formatted at 2026-09-24 10:07:54 on dist-test-slave-s3n3"
server_key: "47cf83ddae9b64fdd0d541635a58c882"
server_key_iv: "1ca0b04d9a42b4a3dd8c7937cb462586"
server_key_version: "encryptionkey@0"
I20260924 10:07:54.306782 10359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:07:54.310323 10373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:54.311854 10359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 10:07:54.312033 10359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "48ff35ff14744390aa0544449c34ff04"
format_stamp: "Formatted at 2026-09-24 10:07:54 on dist-test-slave-s3n3"
server_key: "47cf83ddae9b64fdd0d541635a58c882"
server_key_iv: "1ca0b04d9a42b4a3dd8c7937cb462586"
server_key_version: "encryptionkey@0"
I20260924 10:07:54.312271 10359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:54.332345 10359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:54.336861 10359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:54.337148 10359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:54.338150 10359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:54.339591 10359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:54.339720 10359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:54.339839 10359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:54.339922 10359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:54.375600 10359 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:37695
I20260924 10:07:54.375641 10485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:37695 every 8 connection(s)
I20260924 10:07:54.376497 10359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:07:54.386744 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 10359
I20260924 10:07:54.387006 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:07:54.387843 6155 external_mini_cluster.cc:1428] Setting key 6de5a9f784b14ed7faff6b497072e2a8
Sep 24 10:07:54 dist-test-slave-s3n3 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:54.397477 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:54.379082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:44955 (local address 127.6.2.254:45279)
0924 10:07:54.379523 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:54.379528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:54.380477 (+ 949us) server_negotiation.cc:407] Connection header received
0924 10:07:54.382355 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:54.382361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:54.382451 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:54.382568 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:54.384177 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:54.385016 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:54.386810 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:54.387293 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:54.387404 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:54.391443 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:54.391452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:54.391456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:54.391498 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:54.394205 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:54.395115 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:54.395119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:54.395122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:54.395243 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:54.395720 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:54.395724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:54.395726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:54.395984 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:54.396141 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:54.396875 (+ 734us) server_negotiation.cc:299] Negotiation successful
0924 10:07:54.397101 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":209,"threads_started":1}
I20260924 10:07:54.398516 10489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:54.379611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:45279 (local address 127.6.2.193:44955)
0924 10:07:54.380298 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:54.380341 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:54.382075 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:54.382802 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:54.382812 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:54.383427 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:54.384000 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:54.384017 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:54.385192 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:54.385196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:54.386615 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:54.386634 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:54.387532 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:54.387541 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:54.387873 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:54.388626 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:54.388649 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:54.391214 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:54.394358 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:54.394368 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:54.394387 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:54.394947 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:54.395361 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:54.395366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:54.395369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:54.395554 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:54.396131 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:54.396139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:54.396490 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 10:07:54.396761 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 10:07:54.397333 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":288,"threads_started":1}
I20260924 10:07:54.400210 10486 heartbeater.cc:347] Connected to a master server at 127.6.2.254:45279
I20260924 10:07:54.400545 10486 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:54.401340 10486 heartbeater.cc:511] Master 127.6.2.254:45279 requested a full tablet report, sending...
I20260924 10:07:54.403920 10298 ts_manager.cc:194] Registered new tserver with Master: 48ff35ff14744390aa0544449c34ff04 (127.6.2.193:37695)
I20260924 10:07:54.405308 10298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:44955
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:07:54.438235 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:45279
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:54.775311 10492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:54.775609 10492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:54.775767 10492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:54.783274 10492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:54.783396 10492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:54.783568 10492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:07:54.794032 10492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:45279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:54.795483 10492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:54.796838 10492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:54.811256 10498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:54.811316 10497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:54.811321 10500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:54.812606 10492 server_base.cc:1034] running on GCE node
I20260924 10:07:54.813272 10492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:54.814397 10492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:54.815953 10492 hybrid_clock.cc:648] HybridClock initialized: now 1790244474815896 us; error 57 us; skew 500 ppm
I20260924 10:07:54.816354 10492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:54 dist-test-slave-s3n3 krb5kdc[10254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:54.825992 10492 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:07:54.827448 10492 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:54.827601 10492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:54.828001 10492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:54.831164 10492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "9efd81e107fc419b8219fe1c8f86eafc"
format_stamp: "Formatted at 2026-09-24 10:07:54 on dist-test-slave-s3n3"
server_key: "41dcb46c7f82698e4074dc5d7da4394e"
server_key_iv: "3e140c06d947205d8782e35f09b553ea"
server_key_version: "encryptionkey@0"
I20260924 10:07:54.832036 10492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "9efd81e107fc419b8219fe1c8f86eafc"
format_stamp: "Formatted at 2026-09-24 10:07:54 on dist-test-slave-s3n3"
server_key: "41dcb46c7f82698e4074dc5d7da4394e"
server_key_iv: "3e140c06d947205d8782e35f09b553ea"
server_key_version: "encryptionkey@0"
I20260924 10:07:54.837433 10492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:07:54.841477 10506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:54.842844 10492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:07:54.843027 10492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "9efd81e107fc419b8219fe1c8f86eafc"
format_stamp: "Formatted at 2026-09-24 10:07:54 on dist-test-slave-s3n3"
server_key: "41dcb46c7f82698e4074dc5d7da4394e"
server_key_iv: "3e140c06d947205d8782e35f09b553ea"
server_key_version: "encryptionkey@0"
I20260924 10:07:54.843322 10492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:54.863955 10492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:54.868309 10492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:54.868649 10492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:54.869688 10492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:54.871286 10492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:54.871424 10492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:54.871572 10492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:54.871665 10492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:54.909315 10492 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:45375
I20260924 10:07:54.909377 10618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:45375 every 8 connection(s)
I20260924 10:07:54.910336 10492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:07:54.920151 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 10492
I20260924 10:07:54.920357 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:07:54.920818 6155 external_mini_cluster.cc:1428] Setting key 6bf69e4655a843a46a5ef677578e1364
Sep 24 10:07:54 dist-test-slave-s3n3 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:54.933771 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:54.913108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:52635 (local address 127.6.2.254:45279)
0924 10:07:54.913586 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:54.913591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:54.914986 (+ 1395us) server_negotiation.cc:407] Connection header received
0924 10:07:54.917134 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:54.917141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:54.917247 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:54.917508 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:54.919114 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:54.919960 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:54.922281 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:54.922861 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:54.922994 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:54.927445 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:54.927455 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:54.927462 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:54.927509 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:54.930267 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:54.931296 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:54.931300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:54.931303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:54.931429 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:54.931950 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:54.931954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:54.931956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:54.932264 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:54.932372 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:54.933317 (+ 945us) server_negotiation.cc:299] Negotiation successful
0924 10:07:54.933484 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":265,"threads_started":1}
I20260924 10:07:54.935371 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:54.913550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:45279 (local address 127.6.2.194:52635)
0924 10:07:54.914616 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:54.914690 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:54.916843 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:54.917746 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:54.917755 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:54.918303 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:54.918900 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:54.918916 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:54.920189 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:54.920194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:54.922118 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:54.922134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:54.923295 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:54.923322 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:54.923672 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:54.924671 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:54.924698 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:54.927169 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:54.930419 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:54.930433 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:54.930455 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:54.931151 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:54.931558 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:54.931567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:54.931573 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:54.931806 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:54.932397 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:54.932408 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:54.932901 (+ 493us) client_negotiation.cc:769] Sending connection context
0924 10:07:54.933204 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 10:07:54.933812 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":425,"threads_started":1}
I20260924 10:07:54.937016 10619 heartbeater.cc:347] Connected to a master server at 127.6.2.254:45279
I20260924 10:07:54.937355 10619 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:54.938045 10619 heartbeater.cc:511] Master 127.6.2.254:45279 requested a full tablet report, sending...
I20260924 10:07:54.939555 10298 ts_manager.cc:194] Registered new tserver with Master: 9efd81e107fc419b8219fe1c8f86eafc (127.6.2.194:45375)
I20260924 10:07:54.940347 10298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:52635
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:07:54.967595 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:45279
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:07:55.294811 10625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:55.295279 10625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:55.295456 10625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:55.303011 10625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:55.303221 10625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:55.303442 10625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:07:55.313393 10625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:45279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:55.314792 10625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:55.316239 10625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:55.330063 10630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:55.330111 10631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:55.330202 10633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:55.331518 10625 server_base.cc:1034] running on GCE node
I20260924 10:07:55.332135 10625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:55.333138 10625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:55.334407 10625 hybrid_clock.cc:648] HybridClock initialized: now 1790244475334353 us; error 75 us; skew 500 ppm
I20260924 10:07:55.334779 10625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:07:55 dist-test-slave-s3n3 krb5kdc[10254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244475, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:07:55.337984 10625 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:07:55.339251 10625 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:55.339349 10625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:55.339709 10625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:55.342756 10625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "c635a59eccca48a9842e603ceb6fa9ab"
format_stamp: "Formatted at 2026-09-24 10:07:55 on dist-test-slave-s3n3"
server_key: "5973827838fa88b16996279e1a3b9d77"
server_key_iv: "e9e9492e288345521b57a1d35f3479e2"
server_key_version: "encryptionkey@0"
I20260924 10:07:55.343648 10625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "c635a59eccca48a9842e603ceb6fa9ab"
format_stamp: "Formatted at 2026-09-24 10:07:55 on dist-test-slave-s3n3"
server_key: "5973827838fa88b16996279e1a3b9d77"
server_key_iv: "e9e9492e288345521b57a1d35f3479e2"
server_key_version: "encryptionkey@0"
I20260924 10:07:55.348981 10625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:07:55.352497 10639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:55.354147 10625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:55.354333 10625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "c635a59eccca48a9842e603ceb6fa9ab"
format_stamp: "Formatted at 2026-09-24 10:07:55 on dist-test-slave-s3n3"
server_key: "5973827838fa88b16996279e1a3b9d77"
server_key_iv: "e9e9492e288345521b57a1d35f3479e2"
server_key_version: "encryptionkey@0"
I20260924 10:07:55.354557 10625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:55.408440 10486 heartbeater.cc:503] Master 127.6.2.254:45279 was elected leader, sending a full tablet report...
I20260924 10:07:55.411755 10625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:55.416474 10625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:55.416709 10625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:55.417718 10625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:55.419210 10625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:55.419301 10625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:55.419476 10625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:55.419576 10625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:55.454572 10625 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:36355
I20260924 10:07:55.454674 10751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:36355 every 8 connection(s)
I20260924 10:07:55.455505 10625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:07:55.460592 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 10625
I20260924 10:07:55.460807 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:07:55.461231 6155 external_mini_cluster.cc:1428] Setting key 7359a85212d0a29b43bc0db43011b75d
Sep 24 10:07:55 dist-test-slave-s3n3 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244475, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:07:55.476109 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.457969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:45167 (local address 127.6.2.254:45279)
0924 10:07:55.458480 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:55.458484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:55.459541 (+ 1057us) server_negotiation.cc:407] Connection header received
0924 10:07:55.461537 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:55.461543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:55.461656 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:55.461804 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:55.463840 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.464768 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.466337 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.466755 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.466852 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.470264 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:55.470269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:55.470273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:55.470303 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:55.472887 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:55.473686 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:55.473690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:55.473692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:55.473805 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:55.474265 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:55.474268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:55.474270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:55.474518 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:55.474690 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:55.475498 (+ 808us) server_negotiation.cc:299] Negotiation successful
0924 10:07:55.475717 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":224,"threads_started":1}
I20260924 10:07:55.477180 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.458479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:45279 (local address 127.6.2.195:45167)
0924 10:07:55.459366 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:55.459404 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:55.461241 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:55.462110 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:55.462121 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:55.462926 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:55.463653 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:55.463671 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.464928 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.464933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:55.466196 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:55.466209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.466939 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.466944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:55.467219 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.467869 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:07:55.467891 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:55.470072 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:55.473027 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:55.473072 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:55.473085 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:55.473554 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:55.473960 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:07:55.473964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:07:55.473967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:07:55.474156 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:07:55.474674 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:55.474682 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:55.475038 (+ 356us) client_negotiation.cc:769] Sending connection context
0924 10:07:55.475395 (+ 357us) client_negotiation.cc:240] Negotiation successful
0924 10:07:55.475942 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":334,"threads_started":1}
I20260924 10:07:55.478870 10752 heartbeater.cc:347] Connected to a master server at 127.6.2.254:45279
I20260924 10:07:55.479308 10752 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:55.480090 10752 heartbeater.cc:511] Master 127.6.2.254:45279 requested a full tablet report, sending...
I20260924 10:07:55.481957 10298 ts_manager.cc:194] Registered new tserver with Master: c635a59eccca48a9842e603ceb6fa9ab (127.6.2.195:36355)
I20260924 10:07:55.483249 10298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:45167
I20260924 10:07:55.489045 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:07:55.505811 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.493745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.6.2.254:45279)
0924 10:07:55.493912 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:55.493917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:55.494406 (+ 489us) server_negotiation.cc:407] Connection header received
0924 10:07:55.494552 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:55.494560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:55.494641 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:55.494731 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:55.496261 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.497034 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.498235 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.498624 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.498725 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.500978 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:55.500982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:55.500985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:55.501015 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:55.503494 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:55.504140 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:55.504144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:55.504146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:55.504262 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:55.504593 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:55.504597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:55.504599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:55.504935 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:55.505126 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:55.505392 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 10:07:55.505579 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:07:55.509807 10298 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:07:55.516772 10298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:07:55.544247 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.529089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:36355 (local address 127.0.0.1:48612)
0924 10:07:55.531724 (+ 2635us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:55.531772 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:55.532001 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:55.534507 (+ 2506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:55.534514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:55.534544 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:55.534883 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:55.534894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.538028 (+ 3134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.538033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:55.541361 (+ 3328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:55.541372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.543306 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.543312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:55.543496 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.543557 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 10:07:55.543667 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 10:07:55.543863 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2322,"mutex_wait_us":14,"threads_started":1}
I20260924 10:07:55.544504 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.530021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.6.2.194:45375)
0924 10:07:55.530818 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:55.530832 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:55.535651 (+ 4819us) server_negotiation.cc:407] Connection header received
0924 10:07:55.535935 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:55.535943 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:55.536160 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:55.536325 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:55.537259 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.538736 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.541766 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.543105 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.543217 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.543426 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:55.543997 (+ 571us) server_negotiation.cc:299] Negotiation successful
0924 10:07:55.544168 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":374,"threads_started":1}
I20260924 10:07:55.545572 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.529361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:45375 (local address 127.0.0.1:50054)
0924 10:07:55.535524 (+ 6163us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:55.535546 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:55.535751 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:55.536601 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:55.536606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:55.536629 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:55.537086 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:55.537098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.538919 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.538924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:55.541610 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:55.541621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.543621 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.543626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:55.543798 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.543826 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:07:55.544950 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0924 10:07:55.545088 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6027,"mutex_wait_us":24,"thread_start_us":1903,"threads_started":1}
I20260924 10:07:55.546252 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.529454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48612 (local address 127.6.2.195:36355)
0924 10:07:55.533978 (+ 4524us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:55.533985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:55.534023 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:07:55.534098 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:55.534104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:55.534295 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:55.534492 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:55.535153 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.537860 (+ 2707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.541521 (+ 3661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.543097 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.545408 (+ 2311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.545653 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:55.545721 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 10:07:55.545906 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4363,"thread_start_us":4198,"threads_started":1}
I20260924 10:07:55.549778 10554 tablet_service.cc:1467] Processing CreateTablet for tablet daec6a5600df48a69b186d4b87729df8 (DEFAULT_TABLE table=test-table [id=a51c60f152684ef8a5fbb82c39b41902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:55.549778 10687 tablet_service.cc:1467] Processing CreateTablet for tablet daec6a5600df48a69b186d4b87729df8 (DEFAULT_TABLE table=test-table [id=a51c60f152684ef8a5fbb82c39b41902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:55.551298 10687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daec6a5600df48a69b186d4b87729df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:55.551599 10554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daec6a5600df48a69b186d4b87729df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:55.551884 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.532926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:37695 (local address 127.0.0.1:48200)
0924 10:07:55.544764 (+ 11838us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:55.544783 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:55.545699 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:55.546611 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:55.546617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:55.546643 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:55.546827 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:55.546838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.548490 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.548495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:55.549619 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:55.549630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.551149 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.551157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:55.551323 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.551346 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:07:55.551461 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 10:07:55.551610 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11718,"thread_start_us":2269,"threads_started":1}
I20260924 10:07:55.552407 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.533521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.6.2.193:37695)
0924 10:07:55.544592 (+ 11071us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:55.544606 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:55.544899 (+ 293us) server_negotiation.cc:407] Connection header received
0924 10:07:55.545907 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:55.545916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:55.546195 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:55.546379 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:55.547011 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.548354 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.549779 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.550799 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.550945 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.551202 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:55.551831 (+ 629us) server_negotiation.cc:299] Negotiation successful
0924 10:07:55.552025 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10841,"thread_start_us":11108,"threads_started":1}
I20260924 10:07:55.555215 10421 tablet_service.cc:1467] Processing CreateTablet for tablet daec6a5600df48a69b186d4b87729df8 (DEFAULT_TABLE table=test-table [id=a51c60f152684ef8a5fbb82c39b41902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:55.556360 10421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daec6a5600df48a69b186d4b87729df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:55.566725 10771 tablet_bootstrap.cc:492] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab: Bootstrap starting.
I20260924 10:07:55.570488 10773 tablet_bootstrap.cc:492] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc: Bootstrap starting.
I20260924 10:07:55.572160 10771 tablet_bootstrap.cc:654] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:55.572606 10772 tablet_bootstrap.cc:492] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04: Bootstrap starting.
I20260924 10:07:55.573782 10771 log.cc:824] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:55.574030 10773 tablet_bootstrap.cc:654] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:55.575311 10773 log.cc:824] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:55.576133 10772 tablet_bootstrap.cc:654] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:55.576974 10771 tablet_bootstrap.cc:492] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab: No bootstrap required, opened a new log
I20260924 10:07:55.577252 10771 ts_tablet_manager.cc:1397] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 10:07:55.577286 10772 log.cc:824] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:55.578233 10773 tablet_bootstrap.cc:492] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc: No bootstrap required, opened a new log
I20260924 10:07:55.578548 10773 ts_tablet_manager.cc:1397] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260924 10:07:55.580897 10772 tablet_bootstrap.cc:492] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04: No bootstrap required, opened a new log
I20260924 10:07:55.581172 10772 ts_tablet_manager.cc:1397] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 10:07:55.585740 10773 raft_consensus.cc:348] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c635a59eccca48a9842e603ceb6fa9ab" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36355 } } peers { permanent_uuid: "9efd81e107fc419b8219fe1c8f86eafc" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 45375 } } peers { permanent_uuid: "48ff35ff14744390aa0544449c34ff04" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37695 } }
I20260924 10:07:55.586138 10773 raft_consensus.cc:374] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:55.586283 10773 raft_consensus.cc:729] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efd81e107fc419b8219fe1c8f86eafc, State: Initialized, Role: FOLLOWER
I20260924 10:07:55.586911 10773 consensus_queue.cc:260] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c635a59eccca48a9842e603ceb6fa9ab" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36355 } } peers { permanent_uuid: "9efd81e107fc419b8219fe1c8f86eafc" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 45375 } } peers { permanent_uuid: "48ff35ff14744390aa0544449c34ff04" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37695 } }
I20260924 10:07:55.588124 10619 heartbeater.cc:503] Master 127.6.2.254:45279 was elected leader, sending a full tablet report...
I20260924 10:07:55.588320 10771 raft_consensus.cc:348] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c635a59eccca48a9842e603ceb6fa9ab" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36355 } } peers { permanent_uuid: "9efd81e107fc419b8219fe1c8f86eafc" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 45375 } } peers { permanent_uuid: "48ff35ff14744390aa0544449c34ff04" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37695 } }
I20260924 10:07:55.588727 10771 raft_consensus.cc:374] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:55.588795 10773 ts_tablet_manager.cc:1428] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260924 10:07:55.588814 10771 raft_consensus.cc:729] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c635a59eccca48a9842e603ceb6fa9ab, State: Initialized, Role: FOLLOWER
I20260924 10:07:55.589457 10771 consensus_queue.cc:260] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c635a59eccca48a9842e603ceb6fa9ab" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36355 } } peers { permanent_uuid: "9efd81e107fc419b8219fe1c8f86eafc" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 45375 } } peers { permanent_uuid: "48ff35ff14744390aa0544449c34ff04" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37695 } }
I20260924 10:07:55.590596 10752 heartbeater.cc:503] Master 127.6.2.254:45279 was elected leader, sending a full tablet report...
I20260924 10:07:55.591425 10771 ts_tablet_manager.cc:1428] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
I20260924 10:07:55.592315 10772 raft_consensus.cc:348] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c635a59eccca48a9842e603ceb6fa9ab" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36355 } } peers { permanent_uuid: "9efd81e107fc419b8219fe1c8f86eafc" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 45375 } } peers { permanent_uuid: "48ff35ff14744390aa0544449c34ff04" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37695 } }
I20260924 10:07:55.592803 10772 raft_consensus.cc:374] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:55.592928 10772 raft_consensus.cc:729] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ff35ff14744390aa0544449c34ff04, State: Initialized, Role: FOLLOWER
I20260924 10:07:55.593636 10772 consensus_queue.cc:260] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c635a59eccca48a9842e603ceb6fa9ab" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36355 } } peers { permanent_uuid: "9efd81e107fc419b8219fe1c8f86eafc" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 45375 } } peers { permanent_uuid: "48ff35ff14744390aa0544449c34ff04" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37695 } }
I20260924 10:07:55.595396 10772 ts_tablet_manager.cc:1428] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260924 10:07:55.628851 10487 tablet.cc:2369] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:07:55.662775 10620 tablet.cc:2369] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:07:55.707176 10753 tablet.cc:2369] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:07:55.762830 10777 raft_consensus.cc:482] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:07:55.763180 10777 raft_consensus.cc:504] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c635a59eccca48a9842e603ceb6fa9ab" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36355 } } peers { permanent_uuid: "9efd81e107fc419b8219fe1c8f86eafc" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 45375 } } peers { permanent_uuid: "48ff35ff14744390aa0544449c34ff04" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37695 } }
I20260924 10:07:55.764886 10777 leader_election.cc:288] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c635a59eccca48a9842e603ceb6fa9ab (127.6.2.195:36355), 48ff35ff14744390aa0544449c34ff04 (127.6.2.193:37695)
I20260924 10:07:55.773303 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.765457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:36171 (local address 127.6.2.193:37695)
0924 10:07:55.765710 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:55.765719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:55.766344 (+ 625us) server_negotiation.cc:407] Connection header received
0924 10:07:55.766672 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:55.766678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:55.766755 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:55.766842 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:55.767606 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.768328 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.770939 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.772175 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.772313 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.772416 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:55.772919 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 10:07:55.773051 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 10:07:55.773303 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.765278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:49189 (local address 127.6.2.195:36355)
0924 10:07:55.765573 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:55.765579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:55.766324 (+ 745us) server_negotiation.cc:407] Connection header received
0924 10:07:55.766524 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:55.766531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:55.766634 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:55.766767 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:07:55.767453 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.768211 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.770939 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.772179 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.772313 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.772416 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:55.772919 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 10:07:55.773061 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 10:07:55.773545 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.765241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:37695 (local address 127.6.2.194:36171)
0924 10:07:55.766225 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:55.766240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:55.766470 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:55.766957 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:55.766964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:55.766986 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:55.767439 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:55.767449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.768471 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.768476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:55.770759 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:55.770774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.772474 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.772480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:55.772725 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.772787 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 10:07:55.772872 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 10:07:55.773071 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":262,"thread_start_us":543,"threads_started":1}
I20260924 10:07:55.773593 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.765092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:36355 (local address 127.6.2.194:49189)
0924 10:07:55.766165 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:55.766181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:55.766364 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:55.766783 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:55.766788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:55.766812 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:07:55.767255 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:55.767272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.768348 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.768352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:55.770759 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:07:55.770774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.772462 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.772468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:07:55.772662 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.772774 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 10:07:55.772873 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 10:07:55.773070 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":79,"thread_start_us":817,"threads_started":1}
I20260924 10:07:55.774797 10441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daec6a5600df48a69b186d4b87729df8" candidate_uuid: "9efd81e107fc419b8219fe1c8f86eafc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ff35ff14744390aa0544449c34ff04" is_pre_election: true
I20260924 10:07:55.774792 10707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daec6a5600df48a69b186d4b87729df8" candidate_uuid: "9efd81e107fc419b8219fe1c8f86eafc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c635a59eccca48a9842e603ceb6fa9ab" is_pre_election: true
I20260924 10:07:55.775399 10707 raft_consensus.cc:2454] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efd81e107fc419b8219fe1c8f86eafc in term 0.
I20260924 10:07:55.775399 10441 raft_consensus.cc:2454] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efd81e107fc419b8219fe1c8f86eafc in term 0.
I20260924 10:07:55.776098 10507 leader_election.cc:302] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48ff35ff14744390aa0544449c34ff04, 9efd81e107fc419b8219fe1c8f86eafc; no voters:
I20260924 10:07:55.776449 10777 raft_consensus.cc:2781] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:07:55.776605 10777 raft_consensus.cc:482] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:07:55.776710 10777 raft_consensus.cc:3037] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:55.778823 10777 raft_consensus.cc:504] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c635a59eccca48a9842e603ceb6fa9ab" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36355 } } peers { permanent_uuid: "9efd81e107fc419b8219fe1c8f86eafc" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 45375 } } peers { permanent_uuid: "48ff35ff14744390aa0544449c34ff04" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37695 } }
I20260924 10:07:55.779495 10777 leader_election.cc:288] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [CANDIDATE]: Term 1 election: Requested vote from peers c635a59eccca48a9842e603ceb6fa9ab (127.6.2.195:36355), 48ff35ff14744390aa0544449c34ff04 (127.6.2.193:37695)
I20260924 10:07:55.779953 10707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daec6a5600df48a69b186d4b87729df8" candidate_uuid: "9efd81e107fc419b8219fe1c8f86eafc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c635a59eccca48a9842e603ceb6fa9ab"
I20260924 10:07:55.780002 10441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daec6a5600df48a69b186d4b87729df8" candidate_uuid: "9efd81e107fc419b8219fe1c8f86eafc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ff35ff14744390aa0544449c34ff04"
I20260924 10:07:55.780143 10707 raft_consensus.cc:3037] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:55.780267 10441 raft_consensus.cc:3037] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:55.782361 10707 raft_consensus.cc:2454] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efd81e107fc419b8219fe1c8f86eafc in term 1.
I20260924 10:07:55.782379 10441 raft_consensus.cc:2454] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efd81e107fc419b8219fe1c8f86eafc in term 1.
I20260924 10:07:55.782781 10509 leader_election.cc:302] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9efd81e107fc419b8219fe1c8f86eafc, c635a59eccca48a9842e603ceb6fa9ab; no voters:
I20260924 10:07:55.783146 10777 raft_consensus.cc:2781] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:55.783598 10777 raft_consensus.cc:686] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [term 1 LEADER]: Becoming Leader. State: Replica: 9efd81e107fc419b8219fe1c8f86eafc, State: Running, Role: LEADER
I20260924 10:07:55.784099 10777 consensus_queue.cc:237] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c635a59eccca48a9842e603ceb6fa9ab" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36355 } } peers { permanent_uuid: "9efd81e107fc419b8219fe1c8f86eafc" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 45375 } } peers { permanent_uuid: "48ff35ff14744390aa0544449c34ff04" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37695 } }
I20260924 10:07:55.788867 10298 catalog_manager.cc:5495] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9efd81e107fc419b8219fe1c8f86eafc (127.6.2.194). New cstate: current_term: 1 leader_uuid: "9efd81e107fc419b8219fe1c8f86eafc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c635a59eccca48a9842e603ceb6fa9ab" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9efd81e107fc419b8219fe1c8f86eafc" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 45375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48ff35ff14744390aa0544449c34ff04" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37695 } health_report { overall_health: UNKNOWN } } }
I20260924 10:07:55.836434 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.831018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.6.2.194:45375)
0924 10:07:55.831267 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:55.831274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:55.831381 (+ 107us) server_negotiation.cc:407] Connection header received
0924 10:07:55.831635 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:55.831639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:55.831713 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:55.831868 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:07:55.832240 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.832972 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.834297 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.834707 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.834837 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.835349 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:07:55.835849 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:07:55.836038 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:55.836087 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 10:07:55.836166 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 10:07:55.844795 10707 raft_consensus.cc:1261] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab [term 1 FOLLOWER]: Refusing update from remote peer 9efd81e107fc419b8219fe1c8f86eafc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:55.844795 10441 raft_consensus.cc:1261] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04 [term 1 FOLLOWER]: Refusing update from remote peer 9efd81e107fc419b8219fe1c8f86eafc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:55.845893 10782 consensus_queue.cc:1035] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c635a59eccca48a9842e603ceb6fa9ab" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:55.846263 10777 consensus_queue.cc:1035] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ff35ff14744390aa0544449c34ff04" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:55.858292 10785 mvcc.cc:205] Tried to move back new op lower bound from 7332841373049229312 to 7332841372815609856. Current Snapshot: MvccSnapshot[applied={T|T < 7332841373049229312}]
I20260924 10:07:55.871866 10297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:55.872169 10297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:55.875748 10297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae42931870ab4ff6ab6ee41f79255621: Sending DeleteTablet for 3 replicas of tablet daec6a5600df48a69b186d4b87729df8
I20260924 10:07:55.876868 10554 tablet_service.cc:1514] Processing DeleteTablet for tablet daec6a5600df48a69b186d4b87729df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:55 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:50054
I20260924 10:07:55.877063 10687 tablet_service.cc:1514] Processing DeleteTablet for tablet daec6a5600df48a69b186d4b87729df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:55 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:48612
I20260924 10:07:55.877084 10421 tablet_service.cc:1514] Processing DeleteTablet for tablet daec6a5600df48a69b186d4b87729df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:55 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:48200
I20260924 10:07:55.879365 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 10359
I20260924 10:07:55.889622 10797 tablet_replica.cc:330] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab: stopping tablet replica
I20260924 10:07:55.890130 10797 raft_consensus.cc:2229] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:55.890466 10795 tablet_replica.cc:330] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc: stopping tablet replica
I20260924 10:07:55.890692 10797 raft_consensus.cc:2258] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:55.890913 10795 raft_consensus.cc:2229] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:07:55.891577 10795 raft_consensus.cc:2258] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:55.892437 10796 tablet_replica.cc:330] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04: stopping tablet replica
I20260924 10:07:55.892822 10796 raft_consensus.cc:2229] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:55.893343 10796 raft_consensus.cc:2258] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:55.894062 10797 ts_tablet_manager.cc:1905] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:55.895268 10795 ts_tablet_manager.cc:1905] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:55.899303 10796 ts_tablet_manager.cc:1905] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 10:07:55 dist-test-slave-s3n3 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:07:55.901883 10797 ts_tablet_manager.cc:1918] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:07:55.902052 10797 log.cc:1196] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/daec6a5600df48a69b186d4b87729df8
I20260924 10:07:55.902159 10795 ts_tablet_manager.cc:1918] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:07:55.902295 10795 log.cc:1196] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/daec6a5600df48a69b186d4b87729df8
I20260924 10:07:55.902567 10797 ts_tablet_manager.cc:1939] T daec6a5600df48a69b186d4b87729df8 P c635a59eccca48a9842e603ceb6fa9ab: Deleting consensus metadata
I20260924 10:07:55.902814 10795 ts_tablet_manager.cc:1939] T daec6a5600df48a69b186d4b87729df8 P 9efd81e107fc419b8219fe1c8f86eafc: Deleting consensus metadata
I20260924 10:07:55.904438 10284 catalog_manager.cc:4841] TS c635a59eccca48a9842e603ceb6fa9ab (127.6.2.195:36355): tablet daec6a5600df48a69b186d4b87729df8 (table test-table [id=a51c60f152684ef8a5fbb82c39b41902]) successfully deleted
I20260924 10:07:55.904956 10283 catalog_manager.cc:4841] TS 9efd81e107fc419b8219fe1c8f86eafc (127.6.2.194:45375): tablet daec6a5600df48a69b186d4b87729df8 (table test-table [id=a51c60f152684ef8a5fbb82c39b41902]) successfully deleted
I20260924 10:07:55.907377 10796 ts_tablet_manager.cc:1918] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:07:55.907553 10796 log.cc:1196] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/daec6a5600df48a69b186d4b87729df8
I20260924 10:07:55.908124 10796 ts_tablet_manager.cc:1939] T daec6a5600df48a69b186d4b87729df8 P 48ff35ff14744390aa0544449c34ff04: Deleting consensus metadata
I20260924 10:07:55.910238 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:55.883697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.6.2.193:37695)
0924 10:07:55.883923 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:55.883932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:55.889806 (+ 5874us) server_negotiation.cc:407] Connection header received
0924 10:07:55.890015 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:55.890022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:55.890117 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:55.890235 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:55.892846 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.894026 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.896056 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:55.896661 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:55.896761 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:55.902451 (+ 5690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:55.902465 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:55.902483 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:55.902556 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:55.906668 (+ 4112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:55.907473 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:55.907484 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:55.907491 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:55.907735 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:55.908145 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:55.908150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:55.908153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:55.908774 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:55.909093 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:55.909667 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 10:07:55.909881 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:07:55.910740 10282 catalog_manager.cc:4841] TS 48ff35ff14744390aa0544449c34ff04 (127.6.2.193:37695): tablet daec6a5600df48a69b186d4b87729df8 (table test-table [id=a51c60f152684ef8a5fbb82c39b41902]) successfully deleted
I20260924 10:07:55.915414 10481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:56.015152 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 10359
I20260924 10:07:56.035826 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 10492
Sep 24 10:07:56 dist-test-slave-s3n3 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:07:56.052716 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:56.036798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.6.2.194:45375)
0924 10:07:56.037007 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:56.037013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:56.037034 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:07:56.037101 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:56.037106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:56.037183 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:56.037303 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:56.038476 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:56.039762 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:56.041031 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:56.041710 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:56.041840 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:56.045985 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:56.046004 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:56.046025 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:56.046074 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:56.049658 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:56.050399 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:56.050407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:56.050409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:56.050544 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:56.050945 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:56.050951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:56.050954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:56.051506 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:56.051862 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:56.052126 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 10:07:56.052348 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:07:56.053431 10614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:56.167098 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 10492
I20260924 10:07:56.188761 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 10625
Sep 24 10:07:56 dist-test-slave-s3n3 krb5kdc[10254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:07:56.204188 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:56.189595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48626 (local address 127.6.2.195:36355)
0924 10:07:56.189769 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:56.189774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:56.189810 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:07:56.189973 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:56.189977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:56.190040 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:56.190128 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:56.191373 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:56.192200 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:56.193383 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:56.193775 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:56.193869 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:56.198842 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:56.198865 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:56.198883 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:07:56.198923 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:56.201522 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:56.202237 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:56.202245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:56.202247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:56.202389 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:07:56.202758 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:07:56.202762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:07:56.202764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:07:56.203257 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:56.203489 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:56.203763 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 10:07:56.203918 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:07:56.204792 10747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:56.293666 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 10625
I20260924 10:07:56.310487 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 10267
I20260924 10:07:56.311822 10328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:56.419731 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 10267
2026-09-24T10:07:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4434 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T10:07:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:56Z Disabled control of system clock
I20260924 10:07:56.491328 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:46631
--webserver_interface=127.6.2.254
--webserver_port=0
--builtin_ntp_servers=127.6.2.212:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:46631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 10:07:56.814460 10806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:56.814767 10806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:56.814917 10806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:56.822362 10806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:56.822481 10806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:56.822556 10806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:56.822649 10806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:56.822755 10806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:07:56.832624 10806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:46631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:46631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:56.834009 10806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:56.835959 10806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:56.847689 10811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:56.847780 10814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:56.847818 10812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:56.848652 10806 server_base.cc:1034] running on GCE node
I20260924 10:07:56.849450 10806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:56.850798 10806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:56.852013 10806 hybrid_clock.cc:648] HybridClock initialized: now 1790244476851962 us; error 50 us; skew 500 ppm
I20260924 10:07:56.852396 10806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:07:56.854699 10806 webserver.cc:460] Webserver started at http://127.6.2.254:36677/ using document root <none> and password file <none>
I20260924 10:07:56.855528 10806 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:56.855646 10806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:56.855965 10806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:56.858886 10806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "654e3b2d0349440c8ade08eb858a68a0"
format_stamp: "Formatted at 2026-09-24 10:07:56 on dist-test-slave-s3n3"
server_key: "d170a69c446efb2e1122d0c9be68df4f"
server_key_iv: "e9b3744c50e0e2c8132e253bb88052cf"
server_key_version: "encryptionkey@0"
I20260924 10:07:56.859704 10806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "654e3b2d0349440c8ade08eb858a68a0"
format_stamp: "Formatted at 2026-09-24 10:07:56 on dist-test-slave-s3n3"
server_key: "d170a69c446efb2e1122d0c9be68df4f"
server_key_iv: "e9b3744c50e0e2c8132e253bb88052cf"
server_key_version: "encryptionkey@0"
I20260924 10:07:56.864903 10806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:07:56.868386 10820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:56.869856 10806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:07:56.870040 10806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "654e3b2d0349440c8ade08eb858a68a0"
format_stamp: "Formatted at 2026-09-24 10:07:56 on dist-test-slave-s3n3"
server_key: "d170a69c446efb2e1122d0c9be68df4f"
server_key_iv: "e9b3744c50e0e2c8132e253bb88052cf"
server_key_version: "encryptionkey@0"
I20260924 10:07:56.870275 10806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:56.876305 10806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:56.877101 10806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:56.877394 10806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:56.895804 10806 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:46631
I20260924 10:07:56.895838 10871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:46631 every 8 connection(s)
I20260924 10:07:56.897177 10806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:56.899601 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 10806
I20260924 10:07:56.899791 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:07:56.900442 6155 external_mini_cluster.cc:1428] Setting key fb5a8cb66e44d1043b08fae39442f565
I20260924 10:07:56.901122 10872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:56.908919 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:56.902395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.6.2.254:46631)
0924 10:07:56.903395 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:56.903406 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:56.903445 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:07:56.905471 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:56.905492 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:56.905888 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:56.906217 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:56.907363 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:56.907370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:56.907382 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:56.907396 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:56.907465 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:56.907481 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:56.907672 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:56.907763 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 10:07:56.908035 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":364,"threads_started":1}
I20260924 10:07:56.914098 10872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0: Bootstrap starting.
I20260924 10:07:56.917202 10872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:56.918210 10872 log.cc:824] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:56.920900 10872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0: No bootstrap required, opened a new log
I20260924 10:07:56.928228 10872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654e3b2d0349440c8ade08eb858a68a0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46631 } }
I20260924 10:07:56.928614 10872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:56.928719 10872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654e3b2d0349440c8ade08eb858a68a0, State: Initialized, Role: FOLLOWER
I20260924 10:07:56.929261 10872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654e3b2d0349440c8ade08eb858a68a0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46631 } }
I20260924 10:07:56.929492 10872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:56.929638 10872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:56.929801 10872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:56.932034 10872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654e3b2d0349440c8ade08eb858a68a0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46631 } }
I20260924 10:07:56.932533 10872 leader_election.cc:302] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 654e3b2d0349440c8ade08eb858a68a0; no voters:
I20260924 10:07:56.933153 10872 leader_election.cc:288] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:56.933305 10877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:56.934145 10877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [term 1 LEADER]: Becoming Leader. State: Replica: 654e3b2d0349440c8ade08eb858a68a0, State: Running, Role: LEADER
I20260924 10:07:56.934798 10877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654e3b2d0349440c8ade08eb858a68a0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46631 } }
I20260924 10:07:56.935403 10872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:56.937884 10878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "654e3b2d0349440c8ade08eb858a68a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654e3b2d0349440c8ade08eb858a68a0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46631 } } }
I20260924 10:07:56.938424 10878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:56.938488 10879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 654e3b2d0349440c8ade08eb858a68a0. Latest consensus state: current_term: 1 leader_uuid: "654e3b2d0349440c8ade08eb858a68a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654e3b2d0349440c8ade08eb858a68a0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46631 } } }
I20260924 10:07:56.938860 10879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:56.940770 10884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:56.945948 10884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:56.954923 10884 catalog_manager.cc:1347] Generated new cluster ID: 9bd83c51826d4d08b936f6a1dfde0d7d
I20260924 10:07:56.955050 10884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:56.979744 10884 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:56.980242 10884 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:56.994938 10884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0: Generated new TSK 0
I20260924 10:07:56.995901 10884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:07:57.012290 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46631
--builtin_ntp_servers=127.6.2.212:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 10:07:57.370334 10896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:57.370615 10896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:57.370754 10896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:57.378358 10896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:57.378472 10896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:57.378657 10896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:07:57.388068 10896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:57.389367 10896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:57.391220 10896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:57.404709 10904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:57.404786 10902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:57.404744 10901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:57.406136 10896 server_base.cc:1034] running on GCE node
I20260924 10:07:57.406741 10896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:57.407723 10896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:57.408957 10896 hybrid_clock.cc:648] HybridClock initialized: now 1790244477408924 us; error 52 us; skew 500 ppm
I20260924 10:07:57.409287 10896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:07:57.411643 10896 webserver.cc:460] Webserver started at http://127.6.2.193:45655/ using document root <none> and password file <none>
I20260924 10:07:57.412328 10896 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:57.412458 10896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:57.412773 10896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:57.415702 10896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f"
format_stamp: "Formatted at 2026-09-24 10:07:57 on dist-test-slave-s3n3"
server_key: "967a3f9b3a1e63552f58e7f938069122"
server_key_iv: "40903787055dbb3fe9b578a02477811d"
server_key_version: "encryptionkey@0"
I20260924 10:07:57.416474 10896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f"
format_stamp: "Formatted at 2026-09-24 10:07:57 on dist-test-slave-s3n3"
server_key: "967a3f9b3a1e63552f58e7f938069122"
server_key_iv: "40903787055dbb3fe9b578a02477811d"
server_key_version: "encryptionkey@0"
I20260924 10:07:57.421556 10896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 10:07:57.425242 10910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:57.426556 10896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:07:57.426738 10896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f"
format_stamp: "Formatted at 2026-09-24 10:07:57 on dist-test-slave-s3n3"
server_key: "967a3f9b3a1e63552f58e7f938069122"
server_key_iv: "40903787055dbb3fe9b578a02477811d"
server_key_version: "encryptionkey@0"
I20260924 10:07:57.426973 10896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:57.432725 10896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:57.433516 10896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:57.433784 10896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:57.434770 10896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:57.436261 10896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:57.436378 10896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:57.436506 10896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:57.436597 10896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:57.468912 10896 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:34833
I20260924 10:07:57.468967 11022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:34833 every 8 connection(s)
I20260924 10:07:57.470235 10896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:07:57.475339 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 10896
I20260924 10:07:57.475526 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:07:57.476154 6155 external_mini_cluster.cc:1428] Setting key bc5015b11034497f0572cdd3122cbb08
I20260924 10:07:57.478847 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:57.472802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:33499 (local address 127.6.2.254:46631)
0924 10:07:57.473269 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:57.473274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:57.474541 (+ 1267us) server_negotiation.cc:407] Connection header received
0924 10:07:57.476474 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:57.476479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:57.476553 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:57.476673 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:57.478119 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:57.478124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:57.478127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:57.478141 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:57.478179 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:57.478191 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:57.478287 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:57.478557 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 10:07:57.478617 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":254,"threads_started":1}
I20260924 10:07:57.479095 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46631
--builtin_ntp_servers=127.6.2.212:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 10:07:57.480136 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:57.473382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:46631 (local address 127.6.2.193:33499)
0924 10:07:57.474353 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:57.474394 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:57.476159 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:57.476915 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:57.476936 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:57.477054 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:57.477776 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:07:57.477779 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:57.477815 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:07:57.477945 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:57.478295 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:57.478305 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:57.478313 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:07:57.478598 (+ 285us) client_negotiation.cc:240] Negotiation successful
0924 10:07:57.478857 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":327,"threads_started":1}
I20260924 10:07:57.481724 11023 heartbeater.cc:347] Connected to a master server at 127.6.2.254:46631
I20260924 10:07:57.482066 11023 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:57.482672 11023 heartbeater.cc:511] Master 127.6.2.254:46631 requested a full tablet report, sending...
I20260924 10:07:57.484994 10837 ts_manager.cc:194] Registered new tserver with Master: 13d6ca08a5bf43f7a5cc3c41e12a310f (127.6.2.193:34833)
W20260924 10:07:57.804823 11027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:57.805106 11027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:57.805264 11027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:57.812527 11027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:57.812638 11027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:57.812747 11027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:07:57.821717 11027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:57.822959 11027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:57.824656 11027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:57.837563 11032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:57.837568 11033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:57.837613 11035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:57.838472 11027 server_base.cc:1034] running on GCE node
I20260924 10:07:57.839138 11027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:57.840135 11027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:57.841408 11027 hybrid_clock.cc:648] HybridClock initialized: now 1790244477841338 us; error 53 us; skew 500 ppm
I20260924 10:07:57.841825 11027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:07:57.844262 11027 webserver.cc:460] Webserver started at http://127.6.2.194:39275/ using document root <none> and password file <none>
I20260924 10:07:57.844964 11027 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:57.845049 11027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:57.845387 11027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:57.848637 11027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "b710944b3e984c4a868ca55e02ee07d7"
format_stamp: "Formatted at 2026-09-24 10:07:57 on dist-test-slave-s3n3"
server_key: "99f2e33a598afd71b108e5e9e8024c08"
server_key_iv: "e4c733e870c8f09782949664feff1ce4"
server_key_version: "encryptionkey@0"
I20260924 10:07:57.849462 11027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "b710944b3e984c4a868ca55e02ee07d7"
format_stamp: "Formatted at 2026-09-24 10:07:57 on dist-test-slave-s3n3"
server_key: "99f2e33a598afd71b108e5e9e8024c08"
server_key_iv: "e4c733e870c8f09782949664feff1ce4"
server_key_version: "encryptionkey@0"
I20260924 10:07:57.854945 11027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:07:57.858563 11041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:57.859917 11027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:07:57.860082 11027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "b710944b3e984c4a868ca55e02ee07d7"
format_stamp: "Formatted at 2026-09-24 10:07:57 on dist-test-slave-s3n3"
server_key: "99f2e33a598afd71b108e5e9e8024c08"
server_key_iv: "e4c733e870c8f09782949664feff1ce4"
server_key_version: "encryptionkey@0"
I20260924 10:07:57.860318 11027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:57.866057 11027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:57.866849 11027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:57.867156 11027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:57.868088 11027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:57.869475 11027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:57.869570 11027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:57.869719 11027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:57.869792 11027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:57.902189 11027 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:41867
I20260924 10:07:57.902282 11153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:41867 every 8 connection(s)
I20260924 10:07:57.903592 11027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:07:57.911571 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 11027
I20260924 10:07:57.911779 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:07:57.912427 6155 external_mini_cluster.cc:1428] Setting key b3d8c91073a0d75b9b22cfc3c2286622
I20260924 10:07:57.913601 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:57.906266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:33167 (local address 127.6.2.254:46631)
0924 10:07:57.906492 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:57.906498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:57.908105 (+ 1607us) server_negotiation.cc:407] Connection header received
0924 10:07:57.910573 (+ 2468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:57.910594 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:57.910970 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:57.911225 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:57.912822 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:57.912829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:57.912834 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:57.912861 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:57.912908 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:57.912926 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:57.913094 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:57.913268 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 10:07:57.913337 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 10:07:57.914804 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:57.906832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:46631 (local address 127.6.2.194:33167)
0924 10:07:57.907915 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:57.907965 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:57.910108 (+ 2143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:57.911483 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:57.911495 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:57.911628 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:57.912427 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:07:57.912431 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:57.912489 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:07:57.912598 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:57.913047 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:57.913057 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:57.913065 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:07:57.913323 (+ 258us) client_negotiation.cc:240] Negotiation successful
0924 10:07:57.913587 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":341,"threads_started":1}
I20260924 10:07:57.915544 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46631
--builtin_ntp_servers=127.6.2.212:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 10:07:57.916308 11154 heartbeater.cc:347] Connected to a master server at 127.6.2.254:46631
I20260924 10:07:57.916648 11154 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:57.917363 11154 heartbeater.cc:511] Master 127.6.2.254:46631 requested a full tablet report, sending...
I20260924 10:07:57.918926 10837 ts_manager.cc:194] Registered new tserver with Master: b710944b3e984c4a868ca55e02ee07d7 (127.6.2.194:41867)
W20260924 10:07:58.243520 11157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:58.243796 11157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:58.243891 11157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:58.251142 11157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:58.251252 11157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:58.251361 11157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:07:58.260673 11157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:58.261930 11157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:58.263715 11157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:58.276930 11162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:58.277025 11163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:58.276927 11165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:58.277773 11157 server_base.cc:1034] running on GCE node
I20260924 10:07:58.278434 11157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:58.279450 11157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:58.280649 11157 hybrid_clock.cc:648] HybridClock initialized: now 1790244478280587 us; error 55 us; skew 500 ppm
I20260924 10:07:58.280966 11157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:07:58.283273 11157 webserver.cc:460] Webserver started at http://127.6.2.195:39477/ using document root <none> and password file <none>
I20260924 10:07:58.283968 11157 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:58.284089 11157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:58.284415 11157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:58.287473 11157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "5a775339c1434043935c36bcc055c818"
format_stamp: "Formatted at 2026-09-24 10:07:58 on dist-test-slave-s3n3"
server_key: "3405657d8777f47c056ee780e113b55f"
server_key_iv: "5cef9f88d52cbea4acdde3552b3e3bcf"
server_key_version: "encryptionkey@0"
I20260924 10:07:58.288267 11157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "5a775339c1434043935c36bcc055c818"
format_stamp: "Formatted at 2026-09-24 10:07:58 on dist-test-slave-s3n3"
server_key: "3405657d8777f47c056ee780e113b55f"
server_key_iv: "5cef9f88d52cbea4acdde3552b3e3bcf"
server_key_version: "encryptionkey@0"
I20260924 10:07:58.293468 11157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260924 10:07:58.297318 11171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:58.298766 11157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 10:07:58.298964 11157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "5a775339c1434043935c36bcc055c818"
format_stamp: "Formatted at 2026-09-24 10:07:58 on dist-test-slave-s3n3"
server_key: "3405657d8777f47c056ee780e113b55f"
server_key_iv: "5cef9f88d52cbea4acdde3552b3e3bcf"
server_key_version: "encryptionkey@0"
I20260924 10:07:58.299291 11157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:58.305197 11157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:58.306034 11157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:58.306315 11157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:58.307291 11157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:07:58.308688 11157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:07:58.308815 11157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:58.308944 11157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:07:58.309039 11157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:58.342131 11157 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:44781
I20260924 10:07:58.342216 11283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:44781 every 8 connection(s)
I20260924 10:07:58.343466 11157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:07:58.346467 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 11157
I20260924 10:07:58.346630 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:07:58.347117 6155 external_mini_cluster.cc:1428] Setting key 1e2f4f57ad5dde562f44cdaacb399f75
I20260924 10:07:58.352268 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.346123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:56715 (local address 127.6.2.254:46631)
0924 10:07:58.346343 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:58.346352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:58.348047 (+ 1695us) server_negotiation.cc:407] Connection header received
0924 10:07:58.350019 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:58.350025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:58.350098 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:58.350209 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:58.351514 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:58.351520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:58.351524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:58.351544 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:58.351578 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:58.351594 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:58.351721 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:58.351909 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 10:07:58.351976 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:07:58.353157 11286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.346783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:46631 (local address 127.6.2.195:56715)
0924 10:07:58.347885 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:58.347935 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:58.349718 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:58.350441 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:58.350449 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:58.350546 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:58.351233 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:07:58.351236 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:58.351271 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:07:58.351353 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:58.351694 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:58.351704 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:58.351711 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:07:58.351992 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 10:07:58.352236 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":385,"threads_started":1}
I20260924 10:07:58.354483 11284 heartbeater.cc:347] Connected to a master server at 127.6.2.254:46631
I20260924 10:07:58.354833 11284 heartbeater.cc:464] Registering TS with master...
I20260924 10:07:58.355495 11284 heartbeater.cc:511] Master 127.6.2.254:46631 requested a full tablet report, sending...
I20260924 10:07:58.356949 10837 ts_manager.cc:194] Registered new tserver with Master: 5a775339c1434043935c36bcc055c818 (127.6.2.195:44781)
I20260924 10:07:58.362426 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:07:58.369680 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.366986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.6.2.254:46631)
0924 10:07:58.367341 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:58.367349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:58.367660 (+ 311us) server_negotiation.cc:407] Connection header received
0924 10:07:58.367891 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:58.367896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:58.367944 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:58.368052 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:58.369020 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:58.369026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:58.369031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:58.369049 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:58.369089 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:58.369102 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:58.369227 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:58.369348 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 10:07:58.369408 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:07:58.373585 10837 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:07:58.380548 10837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:07:58.399144 11300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.392400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:41867 (local address 127.0.0.1:53410)
0924 10:07:58.394350 (+ 1950us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:58.394372 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:58.394492 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:58.396482 (+ 1990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:58.396494 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:58.396595 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:58.397751 (+ 1156us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:07:58.397754 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:58.397782 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:07:58.397860 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:58.398379 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:58.398389 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:58.398396 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:07:58.398574 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 10:07:58.398664 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1763,"thread_start_us":4776,"threads_started":1}
I20260924 10:07:58.400369 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.392898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.6.2.194:41867)
0924 10:07:58.393878 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:58.393889 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:58.394633 (+ 744us) server_negotiation.cc:407] Connection header received
0924 10:07:58.394726 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:58.394737 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:58.395034 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:58.395223 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:58.398042 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:58.398051 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:58.398065 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:58.398082 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:58.398147 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:58.398169 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:58.398325 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:58.399537 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0924 10:07:58.399665 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":704,"threads_started":1}
I20260924 10:07:58.401307 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.393921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.6.2.193:34833)
0924 10:07:58.397747 (+ 3826us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:58.397755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:58.397781 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:07:58.397844 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:58.397851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:58.398059 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:58.398200 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:58.400359 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:58.400373 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:58.400391 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:58.400430 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:58.400504 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:58.400523 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:58.400659 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:58.400886 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 10:07:58.400977 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3654,"thread_start_us":3523,"threads_started":1}
I20260924 10:07:58.401362 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.393184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:34833 (local address 127.0.0.1:43426)
0924 10:07:58.397412 (+ 4228us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:58.397450 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:58.397548 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:58.399362 (+ 1814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:58.399366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:58.399386 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:58.400106 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:07:58.400112 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:58.400127 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:07:58.400171 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:58.400719 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:58.400724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:58.400729 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:07:58.400894 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 10:07:58.400945 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4106,"thread_start_us":1637,"threads_started":1}
I20260924 10:07:58.404006 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.392299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:44781 (local address 127.0.0.1:51636)
0924 10:07:58.393111 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:58.393164 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:58.393323 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:58.401333 (+ 8010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:58.401339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:58.401364 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:58.402030 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:07:58.402033 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:58.402048 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:07:58.402087 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:58.403392 (+ 1305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:58.403397 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:58.403402 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:07:58.403521 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 10:07:58.403569 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":1779,"threads_started":1}
I20260924 10:07:58.404902 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.392486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.6.2.195:44781)
0924 10:07:58.400820 (+ 8334us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:58.400828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:58.400860 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:07:58.400928 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:58.400935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:58.401133 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:58.401803 (+ 670us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:58.403168 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:58.403177 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:58.403189 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:58.403205 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:58.403253 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:58.403269 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:58.404382 (+ 1113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:58.404445 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 10:07:58.404544 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8161,"thread_start_us":8000,"threads_started":1}
I20260924 10:07:58.404716 11089 tablet_service.cc:1467] Processing CreateTablet for tablet f58190e227db426cb879ca06b35cf211 (DEFAULT_TABLE table=test-table [id=caa3f9e02f4e4da28c684b0b0b1e078b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:58.404949 10958 tablet_service.cc:1467] Processing CreateTablet for tablet f58190e227db426cb879ca06b35cf211 (DEFAULT_TABLE table=test-table [id=caa3f9e02f4e4da28c684b0b0b1e078b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:58.406227 11089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58190e227db426cb879ca06b35cf211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:58.406248 10958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58190e227db426cb879ca06b35cf211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:58.408481 11219 tablet_service.cc:1467] Processing CreateTablet for tablet f58190e227db426cb879ca06b35cf211 (DEFAULT_TABLE table=test-table [id=caa3f9e02f4e4da28c684b0b0b1e078b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:07:58.409819 11219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f58190e227db426cb879ca06b35cf211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:58.419245 11302 tablet_bootstrap.cc:492] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7: Bootstrap starting.
I20260924 10:07:58.421216 11303 tablet_bootstrap.cc:492] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f: Bootstrap starting.
I20260924 10:07:58.422876 11302 tablet_bootstrap.cc:654] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:58.424144 11302 log.cc:824] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:58.424224 11304 tablet_bootstrap.cc:492] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818: Bootstrap starting.
I20260924 10:07:58.424747 11303 tablet_bootstrap.cc:654] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:58.426218 11303 log.cc:824] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:58.428077 11304 tablet_bootstrap.cc:654] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:58.429177 11303 tablet_bootstrap.cc:492] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f: No bootstrap required, opened a new log
I20260924 10:07:58.429240 11304 log.cc:824] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:58.429492 11303 ts_tablet_manager.cc:1397] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 10:07:58.429931 11302 tablet_bootstrap.cc:492] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7: No bootstrap required, opened a new log
I20260924 10:07:58.430222 11302 ts_tablet_manager.cc:1397] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260924 10:07:58.432291 11304 tablet_bootstrap.cc:492] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818: No bootstrap required, opened a new log
I20260924 10:07:58.432536 11304 ts_tablet_manager.cc:1397] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260924 10:07:58.438076 11303 raft_consensus.cc:348] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a775339c1434043935c36bcc055c818" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 44781 } } peers { permanent_uuid: "b710944b3e984c4a868ca55e02ee07d7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41867 } } peers { permanent_uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 34833 } }
I20260924 10:07:58.438454 11303 raft_consensus.cc:374] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:58.438562 11303 raft_consensus.cc:729] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d6ca08a5bf43f7a5cc3c41e12a310f, State: Initialized, Role: FOLLOWER
I20260924 10:07:58.439262 11303 consensus_queue.cc:260] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a775339c1434043935c36bcc055c818" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 44781 } } peers { permanent_uuid: "b710944b3e984c4a868ca55e02ee07d7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41867 } } peers { permanent_uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 34833 } }
I20260924 10:07:58.440366 11023 heartbeater.cc:503] Master 127.6.2.254:46631 was elected leader, sending a full tablet report...
I20260924 10:07:58.441075 11303 ts_tablet_manager.cc:1428] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f: Time spent starting tablet: real 0.011s user 0.013s sys 0.000s
I20260924 10:07:58.441208 11302 raft_consensus.cc:348] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a775339c1434043935c36bcc055c818" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 44781 } } peers { permanent_uuid: "b710944b3e984c4a868ca55e02ee07d7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41867 } } peers { permanent_uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 34833 } }
I20260924 10:07:58.441636 11302 raft_consensus.cc:374] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:58.441735 11302 raft_consensus.cc:729] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b710944b3e984c4a868ca55e02ee07d7, State: Initialized, Role: FOLLOWER
I20260924 10:07:58.442348 11302 consensus_queue.cc:260] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a775339c1434043935c36bcc055c818" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 44781 } } peers { permanent_uuid: "b710944b3e984c4a868ca55e02ee07d7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41867 } } peers { permanent_uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 34833 } }
I20260924 10:07:58.443419 11154 heartbeater.cc:503] Master 127.6.2.254:46631 was elected leader, sending a full tablet report...
I20260924 10:07:58.444077 11302 ts_tablet_manager.cc:1428] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 10:07:58.442907 11304 raft_consensus.cc:348] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a775339c1434043935c36bcc055c818" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 44781 } } peers { permanent_uuid: "b710944b3e984c4a868ca55e02ee07d7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41867 } } peers { permanent_uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 34833 } }
I20260924 10:07:58.445966 11304 raft_consensus.cc:374] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:58.446098 11304 raft_consensus.cc:729] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a775339c1434043935c36bcc055c818, State: Initialized, Role: FOLLOWER
I20260924 10:07:58.446705 11304 consensus_queue.cc:260] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a775339c1434043935c36bcc055c818" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 44781 } } peers { permanent_uuid: "b710944b3e984c4a868ca55e02ee07d7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41867 } } peers { permanent_uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 34833 } }
I20260924 10:07:58.448235 11284 heartbeater.cc:503] Master 127.6.2.254:46631 was elected leader, sending a full tablet report...
I20260924 10:07:58.448756 11304 ts_tablet_manager.cc:1428] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
W20260924 10:07:58.472455 11024 tablet.cc:2369] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:07:58.595242 11285 tablet.cc:2369] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:07:58.655822 11155 tablet.cc:2369] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:07:58.717955 11310 raft_consensus.cc:482] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:07:58.718252 11310 raft_consensus.cc:504] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a775339c1434043935c36bcc055c818" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 44781 } } peers { permanent_uuid: "b710944b3e984c4a868ca55e02ee07d7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41867 } } peers { permanent_uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 34833 } }
I20260924 10:07:58.719959 11310 leader_election.cc:288] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b710944b3e984c4a868ca55e02ee07d7 (127.6.2.194:41867), 13d6ca08a5bf43f7a5cc3c41e12a310f (127.6.2.193:34833)
I20260924 10:07:58.723001 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.720650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:43701 (local address 127.6.2.193:34833)
0924 10:07:58.720829 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:58.720833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:58.720851 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:07:58.720985 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:58.720993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:58.721064 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:58.721176 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:58.722184 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:58.722193 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:58.722201 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:58.722225 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:58.722304 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:58.722317 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:58.722408 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:58.722679 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 10:07:58.722740 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 10:07:58.723013 11286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.720398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:34833 (local address 127.6.2.195:43701)
0924 10:07:58.720662 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:58.720687 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:58.720815 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:58.721245 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:58.721250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:58.721276 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:58.721962 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:07:58.721964 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:58.721982 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:07:58.722022 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:58.722483 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:58.722491 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:58.722498 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:07:58.722661 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 10:07:58.722708 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 10:07:58.723884 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.720493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:36673 (local address 127.6.2.194:41867)
0924 10:07:58.720700 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:58.720706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:58.721481 (+ 775us) server_negotiation.cc:407] Connection header received
0924 10:07:58.721653 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:58.721665 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:58.721711 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:58.721819 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:58.722892 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:58.722899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:58.722904 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:58.722928 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:58.722964 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:58.722980 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:58.723154 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:58.723573 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 10:07:58.723640 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 10:07:58.724041 11311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.720412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:41867 (local address 127.6.2.195:36673)
0924 10:07:58.721366 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:58.721380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:07:58.721514 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:07:58.721837 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:07:58.721841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:07:58.721865 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:07:58.722662 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:07:58.722665 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:07:58.722679 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:07:58.722720 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:07:58.723130 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:07:58.723135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:07:58.723142 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:07:58.723563 (+ 421us) client_negotiation.cc:240] Negotiation successful
0924 10:07:58.723605 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":161,"thread_start_us":445,"threads_started":1}
I20260924 10:07:58.724306 10978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58190e227db426cb879ca06b35cf211" candidate_uuid: "5a775339c1434043935c36bcc055c818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f" is_pre_election: true
I20260924 10:07:58.724864 10978 raft_consensus.cc:2454] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a775339c1434043935c36bcc055c818 in term 0.
I20260924 10:07:58.725037 11109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58190e227db426cb879ca06b35cf211" candidate_uuid: "5a775339c1434043935c36bcc055c818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b710944b3e984c4a868ca55e02ee07d7" is_pre_election: true
I20260924 10:07:58.725598 11109 raft_consensus.cc:2454] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a775339c1434043935c36bcc055c818 in term 0.
I20260924 10:07:58.725561 11175 leader_election.cc:302] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d6ca08a5bf43f7a5cc3c41e12a310f, 5a775339c1434043935c36bcc055c818; no voters:
I20260924 10:07:58.726030 11310 raft_consensus.cc:2781] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:07:58.726186 11310 raft_consensus.cc:482] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:07:58.726298 11310 raft_consensus.cc:3037] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:58.728542 11310 raft_consensus.cc:504] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a775339c1434043935c36bcc055c818" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 44781 } } peers { permanent_uuid: "b710944b3e984c4a868ca55e02ee07d7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41867 } } peers { permanent_uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 34833 } }
I20260924 10:07:58.729135 11310 leader_election.cc:288] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [CANDIDATE]: Term 1 election: Requested vote from peers b710944b3e984c4a868ca55e02ee07d7 (127.6.2.194:41867), 13d6ca08a5bf43f7a5cc3c41e12a310f (127.6.2.193:34833)
I20260924 10:07:58.729667 11109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58190e227db426cb879ca06b35cf211" candidate_uuid: "5a775339c1434043935c36bcc055c818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b710944b3e984c4a868ca55e02ee07d7"
I20260924 10:07:58.729667 10978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58190e227db426cb879ca06b35cf211" candidate_uuid: "5a775339c1434043935c36bcc055c818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f"
I20260924 10:07:58.729879 10978 raft_consensus.cc:3037] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:58.729878 11109 raft_consensus.cc:3037] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:58.733207 10978 raft_consensus.cc:2454] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a775339c1434043935c36bcc055c818 in term 1.
I20260924 10:07:58.733206 11109 raft_consensus.cc:2454] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a775339c1434043935c36bcc055c818 in term 1.
I20260924 10:07:58.733692 11175 leader_election.cc:302] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d6ca08a5bf43f7a5cc3c41e12a310f, 5a775339c1434043935c36bcc055c818; no voters:
I20260924 10:07:58.734037 11310 raft_consensus.cc:2781] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:58.734477 11310 raft_consensus.cc:686] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [term 1 LEADER]: Becoming Leader. State: Replica: 5a775339c1434043935c36bcc055c818, State: Running, Role: LEADER
I20260924 10:07:58.735024 11310 consensus_queue.cc:237] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a775339c1434043935c36bcc055c818" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 44781 } } peers { permanent_uuid: "b710944b3e984c4a868ca55e02ee07d7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41867 } } peers { permanent_uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 34833 } }
I20260924 10:07:58.740399 10837 catalog_manager.cc:5495] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a775339c1434043935c36bcc055c818 (127.6.2.195). New cstate: current_term: 1 leader_uuid: "5a775339c1434043935c36bcc055c818" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a775339c1434043935c36bcc055c818" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 44781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b710944b3e984c4a868ca55e02ee07d7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 34833 } health_report { overall_health: UNKNOWN } } }
I20260924 10:07:58.770674 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.768729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51644 (local address 127.6.2.195:44781)
0924 10:07:58.768921 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:58.768926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:58.768944 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:07:58.769093 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:58.769105 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:58.769185 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:58.769273 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:58.770205 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:58.770209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:58.770212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:58.770226 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:58.770253 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:58.770264 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:58.770362 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:58.770444 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 10:07:58.770487 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 10:07:58.779801 10978 raft_consensus.cc:1261] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f [term 1 FOLLOWER]: Refusing update from remote peer 5a775339c1434043935c36bcc055c818: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:58.779801 11109 raft_consensus.cc:1261] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7 [term 1 FOLLOWER]: Refusing update from remote peer 5a775339c1434043935c36bcc055c818: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:07:58.780937 11312 consensus_queue.cc:1035] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d6ca08a5bf43f7a5cc3c41e12a310f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:58.781333 11310 consensus_queue.cc:1035] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b710944b3e984c4a868ca55e02ee07d7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:07:58.792013 11317 mvcc.cc:205] Tried to move back new op lower bound from 7332841385070166016 to 7332841384903168000. Current Snapshot: MvccSnapshot[applied={T|T < 7332841385070166016}]
I20260924 10:07:58.810209 10837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:58.810527 10837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:07:58.814646 10837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 654e3b2d0349440c8ade08eb858a68a0: Sending DeleteTablet for 3 replicas of tablet f58190e227db426cb879ca06b35cf211
I20260924 10:07:58.815855 11219 tablet_service.cc:1514] Processing DeleteTablet for tablet f58190e227db426cb879ca06b35cf211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:58 UTC) from {username='slave'} at 127.0.0.1:51636
I20260924 10:07:58.815928 10958 tablet_service.cc:1514] Processing DeleteTablet for tablet f58190e227db426cb879ca06b35cf211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:58 UTC) from {username='slave'} at 127.0.0.1:43426
I20260924 10:07:58.815928 11089 tablet_service.cc:1514] Processing DeleteTablet for tablet f58190e227db426cb879ca06b35cf211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:07:58 UTC) from {username='slave'} at 127.0.0.1:53410
I20260924 10:07:58.824405 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 10896
I20260924 10:07:58.832176 11324 tablet_replica.cc:330] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7: stopping tablet replica
I20260924 10:07:58.832551 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.826441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.6.2.193:34833)
0924 10:07:58.826680 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:58.826689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:58.826716 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:07:58.826798 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:58.826806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:58.826946 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:58.827094 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:58.831131 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:58.831142 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:58.831157 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:58.831192 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:58.831250 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:58.831272 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:58.831393 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:58.832008 (+ 615us) server_negotiation.cc:299] Negotiation successful
0924 10:07:58.832094 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:07:58.833590 11324 raft_consensus.cc:2229] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:58.834373 11018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:58.834940 11324 raft_consensus.cc:2258] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:58.838824 11324 ts_tablet_manager.cc:1905] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:58.841655 11326 tablet_replica.cc:330] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818: stopping tablet replica
I20260924 10:07:58.842227 11326 raft_consensus.cc:2229] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:07:58.843039 11326 raft_consensus.cc:2258] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:07:58.845294 11326 ts_tablet_manager.cc:1905] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:07:58.850193 11324 ts_tablet_manager.cc:1918] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:07:58.850471 11324 log.cc:1196] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/f58190e227db426cb879ca06b35cf211
I20260924 10:07:58.851148 11324 ts_tablet_manager.cc:1939] T f58190e227db426cb879ca06b35cf211 P b710944b3e984c4a868ca55e02ee07d7: Deleting consensus metadata
I20260924 10:07:58.853202 10822 catalog_manager.cc:4841] TS b710944b3e984c4a868ca55e02ee07d7 (127.6.2.194:41867): tablet f58190e227db426cb879ca06b35cf211 (table test-table [id=caa3f9e02f4e4da28c684b0b0b1e078b]) successfully deleted
I20260924 10:07:58.853328 11326 ts_tablet_manager.cc:1918] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:07:58.853514 11326 log.cc:1196] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/f58190e227db426cb879ca06b35cf211
I20260924 10:07:58.854139 11326 ts_tablet_manager.cc:1939] T f58190e227db426cb879ca06b35cf211 P 5a775339c1434043935c36bcc055c818: Deleting consensus metadata
I20260924 10:07:58.856247 10821 catalog_manager.cc:4841] TS 5a775339c1434043935c36bcc055c818 (127.6.2.195:44781): tablet f58190e227db426cb879ca06b35cf211 (table test-table [id=caa3f9e02f4e4da28c684b0b0b1e078b]) successfully deleted
I20260924 10:07:58.952005 11325 tablet_replica.cc:330] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f: stopping tablet replica
I20260924 10:07:58.952482 11325 raft_consensus.cc:2229] T f58190e227db426cb879ca06b35cf211 P 13d6ca08a5bf43f7a5cc3c41e12a310f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:07:58.952527 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 10896
W20260924 10:07:58.976438 10824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.2.193:34833 (error 108)
I20260924 10:07:58.976665 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 11027
I20260924 10:07:58.980214 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.977823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.6.2.194:41867)
0924 10:07:58.978075 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:58.978084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:58.978111 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:07:58.978196 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:58.978204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:58.978341 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:58.978481 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:58.979503 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:58.979508 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:58.979514 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:58.979532 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:58.979586 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:58.979600 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:58.979747 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:58.979830 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 10:07:58.979892 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 10:07:58.980777 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:58.979707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:34833 (local address 127.0.0.1:43450)
0924 10:07:58.980448 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:58.980584 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":487,"threads_started":1}
I20260924 10:07:58.980934 11149 generic_service.cc:196] Checking for leaks (request via RPC)
W20260924 10:07:58.981108 10824 catalog_manager.cc:4584] TS 13d6ca08a5bf43f7a5cc3c41e12a310f (127.6.2.193:34833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f58190e227db426cb879ca06b35cf211: Network error: Client connection negotiation failed: client connection to 127.6.2.193:34833: connect: Connection refused (error 111)
I20260924 10:07:59.034819 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:59.034253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:34833 (local address 127.0.0.1:43462)
0924 10:07:59.034517 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:59.034649 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 10:07:59.036084 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:59.035615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:34833 (local address 127.0.0.1:43474)
0924 10:07:59.035821 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:59.035905 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 10:07:59.080823 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 11027
I20260924 10:07:59.096921 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:59.096372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:34833 (local address 127.0.0.1:43488)
0924 10:07:59.096608 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:59.096722 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 10:07:59.098191 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:59.097701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:34833 (local address 127.0.0.1:43500)
0924 10:07:59.097899 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:59.098003 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 10:07:59.103596 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 11157
I20260924 10:07:59.106741 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:59.104354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.6.2.195:44781)
0924 10:07:59.104545 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:59.104551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:59.104579 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:07:59.104728 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:59.104735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:59.104801 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:59.104920 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:59.105936 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:59.105942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:59.105947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:59.105967 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:59.106010 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:59.106027 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:59.106198 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:59.106361 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 10:07:59.106420 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 10:07:59.107630 11279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:59.179040 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:59.178516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:34833 (local address 127.0.0.1:43508)
0924 10:07:59.178766 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:59.178870 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 10:07:59.180188 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:59.179747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:34833 (local address 127.0.0.1:43516)
0924 10:07:59.179936 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:59.180029 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 10:07:59.231775 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 11157
I20260924 10:07:59.253036 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 10806
I20260924 10:07:59.254287 10867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:07:59.373317 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:59.372767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:34833 (local address 127.0.0.1:43532)
0924 10:07:59.373022 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 10:07:59.373173 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 10:07:59.373332 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 10806
2026-09-24T10:07:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2955 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 10:07:59.418013 6155 mini_oidc.cc:152] Starting JWKS server
I20260924 10:07:59.420715 6155 webserver.cc:460] Webserver started at https://127.0.0.1:41637/ using document root <none> and password file <none>
I20260924 10:07:59.421039 6155 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:07:59.421761 6155 webserver.cc:460] Webserver started at http://127.0.0.1:39309/ using document root <none> and password file <none>
2026-09-24T10:07:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:07:59Z Disabled control of system clock
I20260924 10:07:59.449887 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36373
--webserver_interface=127.6.2.254
--webserver_port=0
--builtin_ntp_servers=127.6.2.212:34047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:36373
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41637/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/testchainca.cert with env {}
W20260924 10:07:59.785476 11339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:07:59.785802 11339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:07:59.785893 11339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:07:59.793419 11339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:07:59.793556 11339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:07:59.793594 11339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:07:59.793643 11339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:07:59.793678 11339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:07:59.793733 11339 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:07:59.793772 11339 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41637/jwks.json
I20260924 10:07:59.804268 11339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:36373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41637/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:07:59.805502 11339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:07:59.806691 11339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:07:59.818534 11344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:59.818779 11345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:07:59.819151 11347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:07:59.819972 11339 server_base.cc:1034] running on GCE node
I20260924 10:07:59.820786 11339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:07:59.822299 11339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:07:59.823567 11339 hybrid_clock.cc:648] HybridClock initialized: now 1790244479823504 us; error 58 us; skew 500 ppm
I20260924 10:07:59.823987 11339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:07:59.841338 11339 webserver.cc:460] Webserver started at http://127.6.2.254:32933/ using document root <none> and password file <none>
I20260924 10:07:59.842145 11339 fs_manager.cc:359] Metadata directory not provided
I20260924 10:07:59.842248 11339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:07:59.842602 11339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:07:59.845712 11339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "92995d4d57e64df9b9724972fc10e0a2"
format_stamp: "Formatted at 2026-09-24 10:07:59 on dist-test-slave-s3n3"
server_key: "69864c45385ffd5193265d26860cfb7e"
server_key_iv: "3fac218f08ea77ffc15141c7d5a75706"
server_key_version: "encryptionkey@0"
I20260924 10:07:59.846568 11339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "92995d4d57e64df9b9724972fc10e0a2"
format_stamp: "Formatted at 2026-09-24 10:07:59 on dist-test-slave-s3n3"
server_key: "69864c45385ffd5193265d26860cfb7e"
server_key_iv: "3fac218f08ea77ffc15141c7d5a75706"
server_key_version: "encryptionkey@0"
I20260924 10:07:59.851917 11339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 10:07:59.855680 11353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:07:59.857600 11339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:07:59.857822 11339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "92995d4d57e64df9b9724972fc10e0a2"
format_stamp: "Formatted at 2026-09-24 10:07:59 on dist-test-slave-s3n3"
server_key: "69864c45385ffd5193265d26860cfb7e"
server_key_iv: "3fac218f08ea77ffc15141c7d5a75706"
server_key_version: "encryptionkey@0"
I20260924 10:07:59.858057 11339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:07:59.874465 11339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:07:59.875448 11339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:07:59.875744 11339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:07:59.895622 11339 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:36373
I20260924 10:07:59.895643 11404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:36373 every 8 connection(s)
I20260924 10:07:59.897045 11339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:07:59.901075 11405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:07:59.901540 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 11339
I20260924 10:07:59.901755 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:07:59.902570 6155 external_mini_cluster.cc:1428] Setting key 43ac666f1275d77bb90c770cac26d154
I20260924 10:07:59.915463 11405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2: Bootstrap starting.
I20260924 10:07:59.917366 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:07:59.904553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.6.2.254:36373)
0924 10:07:59.905570 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0924 10:07:59.905581 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:07:59.905627 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:07:59.907580 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:07:59.907604 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:07:59.908136 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:07:59.908525 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:07:59.909251 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:59.910549 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:59.912510 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:07:59.913265 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:07:59.913405 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:07:59.914732 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:07:59.914740 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:07:59.914754 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:07:59.915405 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:07:59.915489 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:07:59.915511 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:07:59.915660 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:07:59.915911 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 10:07:59.916345 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":392,"threads_started":1}
I20260924 10:07:59.918869 11405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2: Neither blocks nor log segments found. Creating new log.
I20260924 10:07:59.919972 11405 log.cc:824] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2: Log is configured to *not* fsync() on all Append() calls
I20260924 10:07:59.921818 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:07:59.923615 11405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2: No bootstrap required, opened a new log
I20260924 10:07:59.931516 11405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92995d4d57e64df9b9724972fc10e0a2" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36373 } }
I20260924 10:07:59.931890 11405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:07:59.931968 11405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92995d4d57e64df9b9724972fc10e0a2, State: Initialized, Role: FOLLOWER
I20260924 10:07:59.932545 11405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92995d4d57e64df9b9724972fc10e0a2" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36373 } }
I20260924 10:07:59.932739 11405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:07:59.932889 11405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:07:59.933089 11405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:07:59.935361 11405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92995d4d57e64df9b9724972fc10e0a2" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36373 } }
I20260924 10:07:59.935885 11405 leader_election.cc:302] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92995d4d57e64df9b9724972fc10e0a2; no voters:
I20260924 10:07:59.936496 11405 leader_election.cc:288] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:07:59.936683 11410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:07:59.937559 11410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [term 1 LEADER]: Becoming Leader. State: Replica: 92995d4d57e64df9b9724972fc10e0a2, State: Running, Role: LEADER
I20260924 10:07:59.938104 11410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92995d4d57e64df9b9724972fc10e0a2" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36373 } }
I20260924 10:07:59.939553 11405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:07:59.941305 11411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92995d4d57e64df9b9724972fc10e0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92995d4d57e64df9b9724972fc10e0a2" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36373 } } }
I20260924 10:07:59.941699 11411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:59.942762 11412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92995d4d57e64df9b9724972fc10e0a2. Latest consensus state: current_term: 1 leader_uuid: "92995d4d57e64df9b9724972fc10e0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92995d4d57e64df9b9724972fc10e0a2" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36373 } } }
I20260924 10:07:59.943187 11412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2 [sys.catalog]: This master's current role is: LEADER
I20260924 10:07:59.944507 11416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:07:59.950304 11416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:07:59.960079 11416 catalog_manager.cc:1347] Generated new cluster ID: 98b25bc92b474b56be9ba8f5604e6aec
I20260924 10:07:59.960256 11416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:07:59.972077 11416 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:07:59.973531 11416 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:07:59.980767 11416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92995d4d57e64df9b9724972fc10e0a2: Generated new TSK 0
I20260924 10:07:59.981681 11416 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 10:08:00.027004 11435 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 10:08:00.027272 11435 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 10:08:00.028383 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:00.003973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.6.2.254:36373)
0924 10:08:00.004190 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:00.004197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:00.004779 (+ 582us) server_negotiation.cc:407] Connection header received
0924 10:08:00.005076 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:00.005082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:00.005185 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:00.005352 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:08:00.006976 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:00.008206 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:00.009946 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:00.010638 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:00.010797 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:00.011588 (+ 791us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:08:00.027679 (+ 16091us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 10:08:00.028075 (+ 396us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46770: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 10:08:00.028517 11408 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46770: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 10:08:00.029810 11434 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:00.003926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36373 (local address 127.0.0.1:46770)
0924 10:08:00.004664 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:00.004681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:00.004839 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:00.005442 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:00.005445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:00.005456 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:08:00.006093 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:00.006103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:00.008603 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:00.008608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:00.009778 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:00.009790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:00.010837 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:00.010917 (+ 80us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:00.011149 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:00.011241 (+ 92us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:08:00.028026 (+ 16785us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 10:08:00.028310 (+ 284us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.2.254:36373: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":408,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.2.254:36373: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 10:08:00.031812 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 11339
I20260924 10:08:00.033073 11400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:00.130774 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 11339
2026-09-24T10:08:00Z chronyd exiting
I20260924 10:08:00.181761 6155 test_util.cc:175] -----------------------------------------------
I20260924 10:08:00.181906 6155 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790244437048964-6155-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (768 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 10:08:00.186264 6155 mini_oidc.cc:152] Starting JWKS server
I20260924 10:08:00.188222 6155 webserver.cc:460] Webserver started at https://127.0.0.1:42731/ using document root <none> and password file <none>
I20260924 10:08:00.188503 6155 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:08:00.189390 6155 webserver.cc:460] Webserver started at http://127.0.0.1:44863/ using document root <none> and password file <none>
2026-09-24T10:08:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:08:00Z Disabled control of system clock
I20260924 10:08:00.218390 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:44723
--webserver_interface=127.6.2.254
--webserver_port=0
--builtin_ntp_servers=127.6.2.212:36251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:44723
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42731/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/testchainca.cert with env {}
W20260924 10:08:00.555979 11442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:00.556311 11442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:00.556401 11442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:00.563894 11442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:00.564018 11442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:00.564054 11442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:00.564105 11442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:00.564147 11442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:08:00.564200 11442 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:08:00.564239 11442 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42731/jwks.json
I20260924 10:08:00.574628 11442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:36251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:44723
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:44723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42731/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:00.576002 11442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:00.577227 11442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:00.589288 11448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:00.589268 11450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:00.589426 11447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:00.590231 11442 server_base.cc:1034] running on GCE node
I20260924 10:08:00.591195 11442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:00.592689 11442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:00.593961 11442 hybrid_clock.cc:648] HybridClock initialized: now 1790244480593914 us; error 35 us; skew 500 ppm
I20260924 10:08:00.594380 11442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:08:00.597338 11442 webserver.cc:460] Webserver started at http://127.6.2.254:41843/ using document root <none> and password file <none>
I20260924 10:08:00.598451 11442 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:00.598619 11442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:00.599013 11442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:00.602104 11442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "b14f1e47941640a699ff7657368e5d8f"
format_stamp: "Formatted at 2026-09-24 10:08:00 on dist-test-slave-s3n3"
server_key: "b1756573b3884b60e226c73058ab16f8"
server_key_iv: "80bdd12cffa1fd166e241d4816dea6e7"
server_key_version: "encryptionkey@0"
I20260924 10:08:00.602921 11442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "b14f1e47941640a699ff7657368e5d8f"
format_stamp: "Formatted at 2026-09-24 10:08:00 on dist-test-slave-s3n3"
server_key: "b1756573b3884b60e226c73058ab16f8"
server_key_iv: "80bdd12cffa1fd166e241d4816dea6e7"
server_key_version: "encryptionkey@0"
I20260924 10:08:00.608217 11442 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:08:00.611809 11456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:00.613205 11442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:08:00.613409 11442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "b14f1e47941640a699ff7657368e5d8f"
format_stamp: "Formatted at 2026-09-24 10:08:00 on dist-test-slave-s3n3"
server_key: "b1756573b3884b60e226c73058ab16f8"
server_key_iv: "80bdd12cffa1fd166e241d4816dea6e7"
server_key_version: "encryptionkey@0"
I20260924 10:08:00.613644 11442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:00.636004 11442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:00.636898 11442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:00.637203 11442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:00.656419 11442 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:44723
I20260924 10:08:00.656441 11507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:44723 every 8 connection(s)
I20260924 10:08:00.657829 11442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:08:00.659556 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 11442
I20260924 10:08:00.659744 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:08:00.660609 6155 external_mini_cluster.cc:1428] Setting key 9b5f4f5999a2614ac80ced1a72813cd2
I20260924 10:08:00.662698 11508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:00.673934 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:00.662519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.6.2.254:44723)
0924 10:08:00.663697 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:00.663714 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:00.663757 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:08:00.665298 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:00.665317 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:00.665771 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:00.666183 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:00.666751 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:00.668136 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:00.669418 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:00.670124 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:00.670311 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:00.671610 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:00.671619 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:00.671634 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:00.672214 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:00.672289 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:00.672311 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:00.672472 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:00.672698 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 10:08:00.673008 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":529,"threads_started":1}
I20260924 10:08:00.674562 11508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f: Bootstrap starting.
I20260924 10:08:00.677690 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:08:00.677999 11508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:00.679139 11508 log.cc:824] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:00.682094 11508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f: No bootstrap required, opened a new log
I20260924 10:08:00.689397 11508 raft_consensus.cc:348] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b14f1e47941640a699ff7657368e5d8f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44723 } }
I20260924 10:08:00.689751 11508 raft_consensus.cc:374] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:00.689884 11508 raft_consensus.cc:729] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14f1e47941640a699ff7657368e5d8f, State: Initialized, Role: FOLLOWER
I20260924 10:08:00.690539 11508 consensus_queue.cc:260] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b14f1e47941640a699ff7657368e5d8f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44723 } }
I20260924 10:08:00.690759 11508 raft_consensus.cc:388] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:08:00.690915 11508 raft_consensus.cc:482] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:08:00.691119 11508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:00.693284 11508 raft_consensus.cc:504] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b14f1e47941640a699ff7657368e5d8f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44723 } }
I20260924 10:08:00.693800 11508 leader_election.cc:302] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b14f1e47941640a699ff7657368e5d8f; no voters:
I20260924 10:08:00.694440 11508 leader_election.cc:288] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:08:00.694643 11513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:00.695657 11513 raft_consensus.cc:686] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [term 1 LEADER]: Becoming Leader. State: Replica: b14f1e47941640a699ff7657368e5d8f, State: Running, Role: LEADER
I20260924 10:08:00.696170 11513 consensus_queue.cc:237] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b14f1e47941640a699ff7657368e5d8f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44723 } }
I20260924 10:08:00.696833 11508 sys_catalog.cc:564] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:08:00.699108 11515 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b14f1e47941640a699ff7657368e5d8f. Latest consensus state: current_term: 1 leader_uuid: "b14f1e47941640a699ff7657368e5d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b14f1e47941640a699ff7657368e5d8f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44723 } } }
I20260924 10:08:00.699584 11515 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:00.699503 11514 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b14f1e47941640a699ff7657368e5d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b14f1e47941640a699ff7657368e5d8f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44723 } } }
I20260924 10:08:00.699816 11514 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:00.701905 11520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:08:00.707425 11520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:08:00.717473 11520 catalog_manager.cc:1347] Generated new cluster ID: 49d9cb8c189c49268239d84c231d371c
I20260924 10:08:00.717607 11520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:08:00.764057 11520 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:08:00.766038 11520 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:08:00.785902 11520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b14f1e47941640a699ff7657368e5d8f: Generated new TSK 0
I20260924 10:08:00.787050 11520 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 10:08:00.819355 11538 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 10:08:00.819489 11538 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 10:08:00.820520 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:00.800270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.6.2.254:44723)
0924 10:08:00.800508 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:00.800518 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:00.800915 (+ 397us) server_negotiation.cc:407] Connection header received
0924 10:08:00.801167 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:00.801171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:00.801248 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:00.801428 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:08:00.802040 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:00.802873 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:00.804916 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:00.805396 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:00.805499 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:00.806131 (+ 632us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:08:00.819975 (+ 13844us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 10:08:00.820278 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 10:08:00.820663 11511 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 10:08:00.820964 11537 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:00.800150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:44723 (local address 127.0.0.1:40860)
0924 10:08:00.800798 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:00.800818 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:00.800977 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:00.801471 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:00.801476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:00.801490 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:08:00.801805 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:00.801818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:00.803012 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:00.803018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:00.804747 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:00.804761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:00.805525 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:00.805576 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:00.805785 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:00.805834 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:08:00.820264 (+ 14430us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 10:08:00.820560 (+ 296us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.2.254:44723: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":343,"threads_started":1}
I20260924 10:08:00.822093 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 11442
I20260924 10:08:00.823480 11503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:00.920791 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 11442
2026-09-24T10:08:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (791 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 10:08:00.976874 6155 mini_oidc.cc:152] Starting JWKS server
I20260924 10:08:00.978883 6155 webserver.cc:460] Webserver started at https://127.0.0.1:36911/ using document root <none> and password file <none>
I20260924 10:08:00.979189 6155 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:08:00.979892 6155 webserver.cc:460] Webserver started at http://127.0.0.1:37739/ using document root <none> and password file <none>
2026-09-24T10:08:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:08:00Z Disabled control of system clock
I20260924 10:08:01.008687 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:41957
--webserver_interface=127.6.2.254
--webserver_port=0
--builtin_ntp_servers=127.6.2.212:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:41957
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36911/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 10:08:01.338249 11545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:01.338546 11545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:01.338645 11545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:01.346004 11545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:01.346122 11545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:01.346159 11545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:01.346210 11545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:01.346253 11545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:08:01.346307 11545 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:08:01.346346 11545 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36911/jwks.json
I20260924 10:08:01.355800 11545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:41957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:41957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36911/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:01.357069 11545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:01.358801 11545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:01.370461 11553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:01.370527 11551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:01.370498 11550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:01.371397 11545 server_base.cc:1034] running on GCE node
I20260924 10:08:01.372206 11545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:01.373721 11545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:01.374977 11545 hybrid_clock.cc:648] HybridClock initialized: now 1790244481374934 us; error 35 us; skew 500 ppm
I20260924 10:08:01.375376 11545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:08:01.377836 11545 webserver.cc:460] Webserver started at http://127.6.2.254:39371/ using document root <none> and password file <none>
I20260924 10:08:01.378578 11545 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:01.378711 11545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:01.379029 11545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:01.381983 11545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "3fa765571a874d1a8cfd915e5bc73c58"
format_stamp: "Formatted at 2026-09-24 10:08:01 on dist-test-slave-s3n3"
server_key: "9d6d12d5a63ca5f7f01f44cd6418bbe8"
server_key_iv: "2d32c8796aa22dbb4496ac76885729e5"
server_key_version: "encryptionkey@0"
I20260924 10:08:01.382790 11545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "3fa765571a874d1a8cfd915e5bc73c58"
format_stamp: "Formatted at 2026-09-24 10:08:01 on dist-test-slave-s3n3"
server_key: "9d6d12d5a63ca5f7f01f44cd6418bbe8"
server_key_iv: "2d32c8796aa22dbb4496ac76885729e5"
server_key_version: "encryptionkey@0"
I20260924 10:08:01.388152 11545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 10:08:01.391837 11559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:01.393165 11545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:08:01.393355 11545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "3fa765571a874d1a8cfd915e5bc73c58"
format_stamp: "Formatted at 2026-09-24 10:08:01 on dist-test-slave-s3n3"
server_key: "9d6d12d5a63ca5f7f01f44cd6418bbe8"
server_key_iv: "2d32c8796aa22dbb4496ac76885729e5"
server_key_version: "encryptionkey@0"
I20260924 10:08:01.393594 11545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:01.416492 11545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:01.417448 11545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:01.417742 11545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:01.440460 11545 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:41957
I20260924 10:08:01.440483 11610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:41957 every 8 connection(s)
I20260924 10:08:01.441885 11545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:08:01.445770 11611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:01.449831 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 11545
I20260924 10:08:01.450126 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:08:01.450819 6155 external_mini_cluster.cc:1428] Setting key b74738ff8c168fddda356ee74e3291c2
I20260924 10:08:01.456482 11611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58: Bootstrap starting.
I20260924 10:08:01.460110 11611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:01.461278 11611 log.cc:824] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:01.464643 11611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58: No bootstrap required, opened a new log
I20260924 10:08:01.464929 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:01.452927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.6.2.254:41957)
0924 10:08:01.453940 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:01.453951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:01.454003 (+ 52us) server_negotiation.cc:407] Connection header received
0924 10:08:01.455717 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:01.455739 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:01.456195 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:01.456503 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:01.457136 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:01.458436 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:01.460342 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:01.461053 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:01.461202 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:01.462470 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:01.462483 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:01.462506 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:01.463208 (+ 702us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:01.463282 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:01.463302 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:01.463388 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:01.463656 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 10:08:01.463997 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":387,"threads_started":1}
I20260924 10:08:01.468746 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:08:01.472657 11611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa765571a874d1a8cfd915e5bc73c58" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41957 } }
I20260924 10:08:01.473002 11611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:01.473069 11611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa765571a874d1a8cfd915e5bc73c58, State: Initialized, Role: FOLLOWER
I20260924 10:08:01.473693 11611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa765571a874d1a8cfd915e5bc73c58" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41957 } }
I20260924 10:08:01.473902 11611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:08:01.474061 11611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:08:01.474277 11611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:01.477041 11611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa765571a874d1a8cfd915e5bc73c58" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41957 } }
I20260924 10:08:01.477609 11611 leader_election.cc:302] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fa765571a874d1a8cfd915e5bc73c58; no voters:
I20260924 10:08:01.478212 11611 leader_election.cc:288] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:08:01.478355 11616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:01.479156 11616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [term 1 LEADER]: Becoming Leader. State: Replica: 3fa765571a874d1a8cfd915e5bc73c58, State: Running, Role: LEADER
I20260924 10:08:01.479740 11616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa765571a874d1a8cfd915e5bc73c58" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41957 } }
I20260924 10:08:01.480389 11611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:08:01.482421 11618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fa765571a874d1a8cfd915e5bc73c58. Latest consensus state: current_term: 1 leader_uuid: "3fa765571a874d1a8cfd915e5bc73c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa765571a874d1a8cfd915e5bc73c58" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41957 } } }
I20260924 10:08:01.482775 11618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:01.484558 11621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:08:01.484648 11617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fa765571a874d1a8cfd915e5bc73c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa765571a874d1a8cfd915e5bc73c58" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41957 } } }
I20260924 10:08:01.485021 11617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:01.493583 11621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:08:01.504374 11621 catalog_manager.cc:1347] Generated new cluster ID: b8d1ab523bfb4d99aca1d7bdb6235713
I20260924 10:08:01.504542 11621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:08:01.516239 11621 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:08:01.517885 11621 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:08:01.529039 11621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fa765571a874d1a8cfd915e5bc73c58: Generated new TSK 0
I20260924 10:08:01.529964 11621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:08:01.562099 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:01.553643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.6.2.254:41957)
0924 10:08:01.553828 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:01.553835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:01.554201 (+ 366us) server_negotiation.cc:407] Connection header received
0924 10:08:01.554421 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:01.554427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:01.554518 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:01.554684 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:08:01.555139 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:01.556285 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:01.557675 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:01.558370 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:01.558505 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:01.559030 (+ 525us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:08:01.561462 (+ 2432us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36911/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 10:08:01.561777 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54442: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36911/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 10:08:01.562237 11614 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54442: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36911/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 10:08:01.563937 11641 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:01.553470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41957 (local address 127.0.0.1:54442)
0924 10:08:01.554096 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:01.554112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:01.554240 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:01.554707 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:01.554710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:01.554720 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:08:01.554959 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:01.554973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:01.556440 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:01.556444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:01.557537 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:01.557548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:01.558525 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:01.558590 (+ 65us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:01.558718 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:01.558750 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:08:01.561685 (+ 2935us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36911/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 10:08:01.563501 (+ 1816us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.2.254:41957: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36911/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":1494,"thread_start_us":337,"threads_started":1}
I20260924 10:08:01.565027 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 11545
I20260924 10:08:01.566210 11606 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 10:08:01.573900 11642 webserver.cc:533] Webserver: sslize failed:
E20260924 10:08:01.574070 11642 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 10:08:01.659907 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 11545
2026-09-24T10:08:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (736 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T10:08:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:08:01Z Disabled control of system clock
I20260924 10:08:01.740507 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:35055
--webserver_interface=127.6.2.254
--webserver_port=0
--builtin_ntp_servers=127.6.2.212:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:35055
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/testchainca.cert with env {}
W20260924 10:08:02.071499 11647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:02.071797 11647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:02.071889 11647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:02.079525 11647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:02.079653 11647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:02.079690 11647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:02.079751 11647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:02.079794 11647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:08:02.079849 11647 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:08:02.079888 11647 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 10:08:02.090435 11647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:35055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:35055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:02.091734 11647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:02.092958 11647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:02.104712 11652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:02.104682 11655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:02.104737 11653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:02.105633 11647 server_base.cc:1034] running on GCE node
I20260924 10:08:02.106408 11647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:02.107928 11647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:02.109174 11647 hybrid_clock.cc:648] HybridClock initialized: now 1790244482109142 us; error 38 us; skew 500 ppm
I20260924 10:08:02.109587 11647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:08:02.111977 11647 webserver.cc:460] Webserver started at http://127.6.2.254:37631/ using document root <none> and password file <none>
I20260924 10:08:02.112679 11647 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:02.112814 11647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:02.113219 11647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:02.116122 11647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "24c6612a3bee4122b79ce793ef40aa34"
format_stamp: "Formatted at 2026-09-24 10:08:02 on dist-test-slave-s3n3"
server_key: "9f250dc8841346dbc042cb585196271b"
server_key_iv: "119c8ae787f568d7e362f83b3e554772"
server_key_version: "encryptionkey@0"
I20260924 10:08:02.116909 11647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "24c6612a3bee4122b79ce793ef40aa34"
format_stamp: "Formatted at 2026-09-24 10:08:02 on dist-test-slave-s3n3"
server_key: "9f250dc8841346dbc042cb585196271b"
server_key_iv: "119c8ae787f568d7e362f83b3e554772"
server_key_version: "encryptionkey@0"
I20260924 10:08:02.121930 11647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 10:08:02.125346 11661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:02.126677 11647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:08:02.126976 11647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "24c6612a3bee4122b79ce793ef40aa34"
format_stamp: "Formatted at 2026-09-24 10:08:02 on dist-test-slave-s3n3"
server_key: "9f250dc8841346dbc042cb585196271b"
server_key_iv: "119c8ae787f568d7e362f83b3e554772"
server_key_version: "encryptionkey@0"
I20260924 10:08:02.127266 11647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:02.145704 11647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:02.146588 11647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:02.146883 11647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:02.165961 11647 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:35055
I20260924 10:08:02.165980 11712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:35055 every 8 connection(s)
I20260924 10:08:02.167363 11647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:08:02.170742 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 11647
I20260924 10:08:02.170960 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:08:02.171869 6155 external_mini_cluster.cc:1428] Setting key b50f27e2ae396cf1ea68e1727bbc0d31
I20260924 10:08:02.172384 11713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:02.187291 11713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34: Bootstrap starting.
I20260924 10:08:02.190913 11713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:02.192248 11713 log.cc:824] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:02.195870 11713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34: No bootstrap required, opened a new log
I20260924 10:08:02.196626 11716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:02.174269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.6.2.254:35055)
0924 10:08:02.180628 (+ 6359us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:02.180650 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:02.183797 (+ 3147us) server_negotiation.cc:407] Connection header received
0924 10:08:02.186223 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:02.186250 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:02.186884 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:02.187332 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:02.187861 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:02.189808 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:02.191202 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:02.192259 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:02.192455 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:02.193788 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:02.193799 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:02.193819 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:02.194512 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:02.194630 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:02.194658 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:02.194787 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:02.195029 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 10:08:02.195379 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5595,"thread_start_us":5298,"threads_started":1}
I20260924 10:08:02.200593 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:08:02.204386 11713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c6612a3bee4122b79ce793ef40aa34" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35055 } }
I20260924 10:08:02.204751 11713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:02.204847 11713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c6612a3bee4122b79ce793ef40aa34, State: Initialized, Role: FOLLOWER
I20260924 10:08:02.205509 11713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c6612a3bee4122b79ce793ef40aa34" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35055 } }
I20260924 10:08:02.205734 11713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:08:02.205870 11713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:08:02.206041 11713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:02.208277 11713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c6612a3bee4122b79ce793ef40aa34" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35055 } }
I20260924 10:08:02.208786 11713 leader_election.cc:302] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24c6612a3bee4122b79ce793ef40aa34; no voters:
I20260924 10:08:02.209419 11713 leader_election.cc:288] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:08:02.209620 11718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:02.211009 11718 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [term 1 LEADER]: Becoming Leader. State: Replica: 24c6612a3bee4122b79ce793ef40aa34, State: Running, Role: LEADER
I20260924 10:08:02.211552 11718 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c6612a3bee4122b79ce793ef40aa34" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35055 } }
I20260924 10:08:02.211908 11713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:08:02.214363 11719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24c6612a3bee4122b79ce793ef40aa34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c6612a3bee4122b79ce793ef40aa34" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35055 } } }
I20260924 10:08:02.214496 11720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24c6612a3bee4122b79ce793ef40aa34. Latest consensus state: current_term: 1 leader_uuid: "24c6612a3bee4122b79ce793ef40aa34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c6612a3bee4122b79ce793ef40aa34" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35055 } } }
I20260924 10:08:02.214919 11719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:02.214946 11720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:02.215797 11727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:08:02.224084 11727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:08:02.239566 11727 catalog_manager.cc:1347] Generated new cluster ID: c1fce309607641b0863a23ee1412e022
I20260924 10:08:02.239727 11727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:08:02.271341 11727 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:08:02.273017 11727 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:08:02.281615 11727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24c6612a3bee4122b79ce793ef40aa34: Generated new TSK 0
I20260924 10:08:02.282646 11727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:08:02.362352 11716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:02.306645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.6.2.254:35055)
0924 10:08:02.306845 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:02.306851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:02.307282 (+ 431us) server_negotiation.cc:407] Connection header received
0924 10:08:02.307500 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:02.307507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:02.307613 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:02.307778 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:08:02.308613 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:02.309755 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:02.311346 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:02.312038 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:02.312165 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:02.312843 (+ 678us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:08:02.361622 (+ 48779us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 10:08:02.362022 (+ 400us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34698: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 10:08:02.362497 11716 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34698: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 10:08:02.362664 11742 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:02.306490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:35055 (local address 127.0.0.1:34698)
0924 10:08:02.307179 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:02.307195 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:02.307321 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:02.307938 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:02.307944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:02.307965 (+ 21us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:08:02.308398 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:02.308410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:02.309990 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:02.309995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:02.311203 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:02.311215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:02.312185 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:02.312235 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:02.312373 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:02.312536 (+ 163us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:08:02.361971 (+ 49435us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 10:08:02.362236 (+ 265us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.2.254:35055: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":371,"threads_started":1}
I20260924 10:08:02.364253 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 11647
I20260924 10:08:02.365516 11708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:02.474905 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 11647
2026-09-24T10:08:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (798 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (413 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (411 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:08:03 dist-test-slave-s3n3 krb5kdc[11757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:08:03 dist-test-slave-s3n3 krb5kdc[11757](info): set up 2 sockets
Sep 24 10:08:03 dist-test-slave-s3n3 krb5kdc[11757](info): commencing operation
krb5kdc: starting...
W20260924 10:08:04.403891 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:08:04 dist-test-slave-s3n3 krb5kdc[11757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:08:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:08:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:08:04.624850 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34929
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:34929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:04.956290 11770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:04.956648 11770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:04.956816 11770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:04.964262 11770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:04.964421 11770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:04.964531 11770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:04.964635 11770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:04.964723 11770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:08:04.974762 11770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:34929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:04.976176 11770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:04.977567 11770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:04.990031 11776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:04.990000 11775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:04.989977 11778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:04.990890 11770 server_base.cc:1034] running on GCE node
I20260924 10:08:04.991793 11770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:04.993310 11770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:04.994555 11770 hybrid_clock.cc:648] HybridClock initialized: now 1790244484994508 us; error 43 us; skew 500 ppm
I20260924 10:08:04.994925 11770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:05 dist-test-slave-s3n3 krb5kdc[11757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244485, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:05.004271 11770 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:08:05.005959 11770 webserver.cc:460] Webserver started at http://127.6.2.254:42063/ using document root <none> and password file <none>
I20260924 10:08:05.006642 11770 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:05.006742 11770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:05.007009 11770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:05.010093 11770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "c5533597096f482a8da25aba4efc371f"
format_stamp: "Formatted at 2026-09-24 10:08:05 on dist-test-slave-s3n3"
server_key: "3f5fd56ce1b8ad726271b7d2fcbb2f3d"
server_key_iv: "cb41ee73031db22dec35657b287355a9"
server_key_version: "encryptionkey@0"
I20260924 10:08:05.010828 11770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "c5533597096f482a8da25aba4efc371f"
format_stamp: "Formatted at 2026-09-24 10:08:05 on dist-test-slave-s3n3"
server_key: "3f5fd56ce1b8ad726271b7d2fcbb2f3d"
server_key_iv: "cb41ee73031db22dec35657b287355a9"
server_key_version: "encryptionkey@0"
I20260924 10:08:05.017140 11770 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 10:08:05.021273 11785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:05.022719 11770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:08:05.022924 11770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "c5533597096f482a8da25aba4efc371f"
format_stamp: "Formatted at 2026-09-24 10:08:05 on dist-test-slave-s3n3"
server_key: "3f5fd56ce1b8ad726271b7d2fcbb2f3d"
server_key_iv: "cb41ee73031db22dec35657b287355a9"
server_key_version: "encryptionkey@0"
I20260924 10:08:05.023226 11770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:05.038089 11770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:05.042548 11770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:05.042850 11770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:05.063642 11770 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:34929
I20260924 10:08:05.063676 11836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:34929 every 8 connection(s)
I20260924 10:08:05.065119 11770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:08:05.067312 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 11770
I20260924 10:08:05.067520 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:08:05.068147 6155 external_mini_cluster.cc:1428] Setting key 1575ff46cb928758485b9df8d6910517
I20260924 10:08:05.069916 11837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 10:08:05 dist-test-slave-s3n3 krb5kdc[11757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:08:05.085264 11837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f: Bootstrap starting.
I20260924 10:08:05.089614 11837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:05.090930 11837 log.cc:824] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:05.093355 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:05.070310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.6.2.254:34929)
0924 10:08:05.071149 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:05.071164 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:05.071215 (+ 51us) server_negotiation.cc:407] Connection header received
0924 10:08:05.073220 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:05.073248 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:05.073787 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:05.074259 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:05.075642 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:05.077712 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:05.078929 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:05.080450 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:05.080661 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:05.084625 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:05.084636 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:05.084658 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:05.084711 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:05.088664 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:05.089389 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:05.089400 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:05.089412 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:05.089633 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:05.090095 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:05.090106 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:05.090109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:05.090830 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:05.091182 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:05.091549 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 10:08:05.092063 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":319,"threads_started":1}
I20260924 10:08:05.094304 11837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f: No bootstrap required, opened a new log
I20260924 10:08:05.101830 11837 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5533597096f482a8da25aba4efc371f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34929 } }
I20260924 10:08:05.102346 11837 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:05.102474 11837 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5533597096f482a8da25aba4efc371f, State: Initialized, Role: FOLLOWER
I20260924 10:08:05.103161 11837 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5533597096f482a8da25aba4efc371f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34929 } }
I20260924 10:08:05.103390 11837 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:08:05.103539 11837 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:08:05.103718 11837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:05.105885 11837 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5533597096f482a8da25aba4efc371f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34929 } }
I20260924 10:08:05.106379 11837 leader_election.cc:302] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5533597096f482a8da25aba4efc371f; no voters:
I20260924 10:08:05.106941 11837 leader_election.cc:288] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:08:05.107167 11842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:05.108469 11842 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [term 1 LEADER]: Becoming Leader. State: Replica: c5533597096f482a8da25aba4efc371f, State: Running, Role: LEADER
I20260924 10:08:05.108989 11842 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5533597096f482a8da25aba4efc371f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34929 } }
I20260924 10:08:05.109336 11837 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:08:05.112052 11844 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5533597096f482a8da25aba4efc371f. Latest consensus state: current_term: 1 leader_uuid: "c5533597096f482a8da25aba4efc371f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5533597096f482a8da25aba4efc371f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34929 } } }
I20260924 10:08:05.112012 11843 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5533597096f482a8da25aba4efc371f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5533597096f482a8da25aba4efc371f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34929 } } }
I20260924 10:08:05.112448 11844 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:05.112448 11843 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:05.114104 11849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:08:05.119306 11849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:08:05.129256 11849 catalog_manager.cc:1347] Generated new cluster ID: e18c2a8e8cb54399b661bf3e9b567110
I20260924 10:08:05.129523 11849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:08:05.159624 11849 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:08:05.161018 11849 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:08:05.171891 11849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5533597096f482a8da25aba4efc371f: Generated new TSK 0
I20260924 10:08:05.172976 11849 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:08:05.235435 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34929
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:05.587900 11863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:05.588220 11863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:05.588387 11863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:05.596282 11863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:05.596451 11863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:05.596632 11863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:08:05.607035 11863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:05.608548 11863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:05.609992 11863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:05.624297 11871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:05.624330 11868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:05.624352 11869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:05.625422 11863 server_base.cc:1034] running on GCE node
I20260924 10:08:05.626039 11863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:05.627128 11863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:05.628340 11863 hybrid_clock.cc:648] HybridClock initialized: now 1790244485628299 us; error 37 us; skew 500 ppm
I20260924 10:08:05.628624 11863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:05 dist-test-slave-s3n3 krb5kdc[11757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244485, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:05.631822 11863 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:08:05.633379 11863 webserver.cc:460] Webserver started at http://127.6.2.193:39955/ using document root <none> and password file <none>
I20260924 10:08:05.634055 11863 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:05.634158 11863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:05.634407 11863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:05.637403 11863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "1104a8dcc0d14c18a85db70eebbd737c"
format_stamp: "Formatted at 2026-09-24 10:08:05 on dist-test-slave-s3n3"
server_key: "326276234fc67daeccd4e9027b8290c4"
server_key_iv: "8bc9f29354da762e50a109de9ca9c887"
server_key_version: "encryptionkey@0"
I20260924 10:08:05.638230 11863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "1104a8dcc0d14c18a85db70eebbd737c"
format_stamp: "Formatted at 2026-09-24 10:08:05 on dist-test-slave-s3n3"
server_key: "326276234fc67daeccd4e9027b8290c4"
server_key_iv: "8bc9f29354da762e50a109de9ca9c887"
server_key_version: "encryptionkey@0"
I20260924 10:08:05.643576 11863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 10:08:05.647195 11878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:05.648648 11863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:08:05.648813 11863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "1104a8dcc0d14c18a85db70eebbd737c"
format_stamp: "Formatted at 2026-09-24 10:08:05 on dist-test-slave-s3n3"
server_key: "326276234fc67daeccd4e9027b8290c4"
server_key_iv: "8bc9f29354da762e50a109de9ca9c887"
server_key_version: "encryptionkey@0"
I20260924 10:08:05.648985 11863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:05.661501 11863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:05.666226 11863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:05.666520 11863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:05.667526 11863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:05.668912 11863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:05.668987 11863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:05.669075 11863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:05.669121 11863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:05.705610 11863 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:42241
I20260924 10:08:05.705662 11990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:42241 every 8 connection(s)
I20260924 10:08:05.706974 11863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:08:05.710310 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 11863
I20260924 10:08:05.710512 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:08:05.711239 6155 external_mini_cluster.cc:1428] Setting key 18485c0965ec5784e6fec32851a8baee
Sep 24 10:08:05 dist-test-slave-s3n3 krb5kdc[11757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244485, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:08:05.731585 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:05.710033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:34773 (local address 127.6.2.254:34929)
0924 10:08:05.710601 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:05.710607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:05.711757 (+ 1150us) server_negotiation.cc:407] Connection header received
0924 10:08:05.713805 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:05.713809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:05.713870 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:05.713958 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:05.716281 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:05.717065 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:05.719891 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:05.720497 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:05.720706 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:05.724380 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:05.724389 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:05.724397 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:05.724443 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:05.728264 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:05.729126 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:05.729131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:05.729135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:05.729274 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:05.729723 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:05.729728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:05.729732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:05.730006 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:05.730167 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:05.730974 (+ 807us) server_negotiation.cc:299] Negotiation successful
0924 10:08:05.731225 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":242,"threads_started":1}
I20260924 10:08:05.732604 11994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:05.710651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34929 (local address 127.6.2.193:34773)
0924 10:08:05.711573 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:05.711626 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:05.713506 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:05.714252 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:05.714266 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:05.715129 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:05.716017 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:05.716045 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:05.717218 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:05.717227 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:05.719705 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:05.719727 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:05.720790 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:05.720796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:05.721020 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:05.721809 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:05.721833 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:05.724113 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:05.728449 (+ 4336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:05.728475 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:05.728494 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:05.728983 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:05.729392 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:05.729398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:05.729401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:05.729592 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:05.730138 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:05.730147 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:05.730557 (+ 410us) client_negotiation.cc:769] Sending connection context
0924 10:08:05.730860 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 10:08:05.731451 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":418,"threads_started":1}
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
I20260924 10:08:05.734231 11991 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34929
I20260924 10:08:05.734555 11991 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:05.735414 11991 heartbeater.cc:511] Master 127.6.2.254:34929 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
I20260924 10:08:05.738581 11802 ts_manager.cc:194] Registered new tserver with Master: 1104a8dcc0d14c18a85db70eebbd737c (127.6.2.193:42241)
I20260924 10:08:05.740743 11802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:34773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:08:05.767438 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34929
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:06.100713 11997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:06.101082 11997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:06.101254 11997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:06.108681 11997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:06.108860 11997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:06.109062 11997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:08:06.119210 11997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:06.120566 11997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:06.121861 11997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:06.135771 12002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:06.135821 12003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:06.135844 12005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:06.136745 11997 server_base.cc:1034] running on GCE node
I20260924 10:08:06.137378 11997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:06.138418 11997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:06.139674 11997 hybrid_clock.cc:648] HybridClock initialized: now 1790244486139606 us; error 52 us; skew 500 ppm
I20260924 10:08:06.140033 11997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:06.143478 11997 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:08:06.145426 11997 webserver.cc:460] Webserver started at http://127.6.2.194:46643/ using document root <none> and password file <none>
I20260924 10:08:06.146193 11997 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:06.146299 11997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:06.146678 11997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:06.149802 11997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "c98191cfea6948008acf1c8c74a118ef"
format_stamp: "Formatted at 2026-09-24 10:08:06 on dist-test-slave-s3n3"
server_key: "b3faeb2d9f4064e98da6cb2228d2d96d"
server_key_iv: "252812defa37dc1a3db6f1fb8add75ca"
server_key_version: "encryptionkey@0"
I20260924 10:08:06.150607 11997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "c98191cfea6948008acf1c8c74a118ef"
format_stamp: "Formatted at 2026-09-24 10:08:06 on dist-test-slave-s3n3"
server_key: "b3faeb2d9f4064e98da6cb2228d2d96d"
server_key_iv: "252812defa37dc1a3db6f1fb8add75ca"
server_key_version: "encryptionkey@0"
I20260924 10:08:06.156318 11997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:08:06.160161 12012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:06.161580 11997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:08:06.161760 11997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "c98191cfea6948008acf1c8c74a118ef"
format_stamp: "Formatted at 2026-09-24 10:08:06 on dist-test-slave-s3n3"
server_key: "b3faeb2d9f4064e98da6cb2228d2d96d"
server_key_iv: "252812defa37dc1a3db6f1fb8add75ca"
server_key_version: "encryptionkey@0"
I20260924 10:08:06.161994 11997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:06.182377 11997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:06.187003 11997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:06.187359 11997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:06.188421 11997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:06.190464 11997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:06.190541 11997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:06.190845 11997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:06.190908 11997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:06.226928 11997 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:43689
I20260924 10:08:06.227036 12124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:43689 every 8 connection(s)
I20260924 10:08:06.228349 11997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:08:06.229399 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 11997
I20260924 10:08:06.229596 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:08:06.230222 6155 external_mini_cluster.cc:1428] Setting key 99d0c107b56a4ec3a78ce10802f8f347
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:08:06.253530 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:06.231570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:58421 (local address 127.6.2.254:34929)
0924 10:08:06.232421 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:06.232427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:06.233343 (+ 916us) server_negotiation.cc:407] Connection header received
0924 10:08:06.236690 (+ 3347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:06.236699 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:06.236792 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:06.236901 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:06.239309 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.240344 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.242375 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.242813 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.242924 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:06.247612 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:06.247618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:06.247622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:06.247656 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:06.250337 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:06.251140 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:06.251148 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:06.251154 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:06.251281 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:06.251728 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:06.251733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:06.251737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:06.252103 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:06.252258 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:06.252962 (+ 704us) server_negotiation.cc:299] Negotiation successful
0924 10:08:06.253189 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717}
I20260924 10:08:06.254739 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:06.232132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34929 (local address 127.6.2.194:58421)
0924 10:08:06.233143 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:06.233202 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:06.236309 (+ 3107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:06.237219 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:06.237236 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:06.238165 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:06.239111 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:06.239143 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.240509 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.240515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:06.242171 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:06.242192 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.243055 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.243087 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:06.243420 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:06.244398 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:06.244437 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:06.247335 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:06.250511 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:06.250522 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:06.250542 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:06.250982 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:06.251398 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:06.251404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:06.251408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:06.251608 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:06.252228 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:06.252236 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:06.252601 (+ 365us) client_negotiation.cc:769] Sending connection context
0924 10:08:06.252876 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 10:08:06.253508 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":443,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
I20260924 10:08:06.256356 12125 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34929
I20260924 10:08:06.256737 12125 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:06.257694 12125 heartbeater.cc:511] Master 127.6.2.254:34929 requested a full tablet report, sending...
I20260924 10:08:06.259403 11802 ts_manager.cc:194] Registered new tserver with Master: c98191cfea6948008acf1c8c74a118ef (127.6.2.194:43689)
I20260924 10:08:06.260659 11802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:58421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:08:06.287410 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34929
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:06.630196 12130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:06.630487 12130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:06.630641 12130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:06.638288 12130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:06.638415 12130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:06.638595 12130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:08:06.648523 12130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:06.649905 12130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:06.651301 12130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:06.665109 12135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:06.665025 12136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:06.665083 12138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:06.665911 12130 server_base.cc:1034] running on GCE node
I20260924 10:08:06.666554 12130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:06.667794 12130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:06.669068 12130 hybrid_clock.cc:648] HybridClock initialized: now 1790244486669025 us; error 73 us; skew 500 ppm
I20260924 10:08:06.669422 12130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:06.672772 12130 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:08:06.674355 12130 webserver.cc:460] Webserver started at http://127.6.2.195:42633/ using document root <none> and password file <none>
I20260924 10:08:06.675050 12130 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:06.675271 12130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:06.675593 12130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:06.678555 12130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "2db0c64847144e518fcb589b39e155f6"
format_stamp: "Formatted at 2026-09-24 10:08:06 on dist-test-slave-s3n3"
server_key: "dd1a14cc089716d09a37a2a47093d426"
server_key_iv: "fd98ca792b217b5cfaabe95be2dcad7f"
server_key_version: "encryptionkey@0"
I20260924 10:08:06.679518 12130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "2db0c64847144e518fcb589b39e155f6"
format_stamp: "Formatted at 2026-09-24 10:08:06 on dist-test-slave-s3n3"
server_key: "dd1a14cc089716d09a37a2a47093d426"
server_key_iv: "fd98ca792b217b5cfaabe95be2dcad7f"
server_key_version: "encryptionkey@0"
I20260924 10:08:06.685727 12130 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260924 10:08:06.689739 12145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:06.691267 12130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 10:08:06.691499 12130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "2db0c64847144e518fcb589b39e155f6"
format_stamp: "Formatted at 2026-09-24 10:08:06 on dist-test-slave-s3n3"
server_key: "dd1a14cc089716d09a37a2a47093d426"
server_key_iv: "fd98ca792b217b5cfaabe95be2dcad7f"
server_key_version: "encryptionkey@0"
I20260924 10:08:06.691730 12130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:06.710245 12130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:06.714782 12130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:06.715031 12130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:06.716317 12130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:06.717803 12130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:06.717876 12130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:06.718007 12130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:06.718058 12130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:06.744267 11991 heartbeater.cc:503] Master 127.6.2.254:34929 was elected leader, sending a full tablet report...
I20260924 10:08:06.754733 12130 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:40271
I20260924 10:08:06.754779 12257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:40271 every 8 connection(s)
I20260924 10:08:06.756088 12130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:08:06.760624 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 12130
I20260924 10:08:06.760790 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:08:06.761267 6155 external_mini_cluster.cc:1428] Setting key f7303ee622bd3cfab01d888e5ab9fe0c
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:08:06.778543 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:06.758913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:34563 (local address 127.6.2.254:34929)
0924 10:08:06.759399 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:06.759404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:06.760257 (+ 853us) server_negotiation.cc:407] Connection header received
0924 10:08:06.762396 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:06.762401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:06.762498 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:06.762626 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:06.764328 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.765233 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.767111 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.767545 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.767695 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:06.771653 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:06.771658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:06.771661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:06.771694 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:06.775035 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:06.775980 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:06.775986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:06.775990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:06.776130 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:06.776596 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:06.776606 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:06.776609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:06.776888 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:06.777069 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:06.777882 (+ 813us) server_negotiation.cc:299] Negotiation successful
0924 10:08:06.778130 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":283,"threads_started":1}
I20260924 10:08:06.779662 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:06.759364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34929 (local address 127.6.2.195:34563)
0924 10:08:06.760092 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:06.760132 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:06.762115 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:06.762880 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:06.762889 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:06.763547 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:06.764141 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:06.764160 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.765417 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.765422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:06.766917 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:06.766931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.767800 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.767807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:06.768066 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:06.768760 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:06.768789 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:06.771369 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:06.775310 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:06.775321 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:06.775341 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:06.775830 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:06.776250 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:06.776256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:06.776260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:06.776458 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:06.777030 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:06.777040 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:06.777444 (+ 404us) client_negotiation.cc:769] Sending connection context
0924 10:08:06.777778 (+ 334us) client_negotiation.cc:240] Negotiation successful
0924 10:08:06.778363 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":293,"threads_started":1}
I20260924 10:08:06.780997 12258 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34929
I20260924 10:08:06.781316 12258 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:06.782105 12258 heartbeater.cc:511] Master 127.6.2.254:34929 requested a full tablet report, sending...
I20260924 10:08:06.783593 11802 ts_manager.cc:194] Registered new tserver with Master: 2db0c64847144e518fcb589b39e155f6 (127.6.2.195:40271)
I20260924 10:08:06.784941 11802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:34563
I20260924 10:08:06.790971 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/test-admin.keytab.
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:06.823204 6155 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 10:08:06.828706 6155 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/krb5cc' of type FILE
W20260924 10:08:06.828802 12266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:06.825964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.6.2.193:42241)
0924 10:08:06.826703 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:06.826712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:06.826739 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:08:06.826809 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:06.826816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:06.827034 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:06.827236 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:06.828320 (+ 1084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60496: BlockingRecv error: recv got EOF from 127.0.0.1:60496 (error 108)
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":368,"threads_started":1}
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:06.830441 6155 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:06.832136 6155 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:08:06.849371 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:06.834016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.6.2.193:42241)
0924 10:08:06.834242 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:06.834250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:06.834376 (+ 126us) server_negotiation.cc:407] Connection header received
0924 10:08:06.834584 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:06.834589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:06.834654 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:06.834745 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:06.835895 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.836979 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.838331 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.838908 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.839028 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:06.843265 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:06.843277 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:06.843306 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:06.843358 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:06.846412 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:06.847207 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:06.847215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:06.847217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:06.847362 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:06.847861 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:06.847865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:06.847867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:06.848248 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:06.848507 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:06.848813 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 10:08:06.849099 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260924 10:08:06.856305 12266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:06.854012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.6.2.193:42241)
0924 10:08:06.854319 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:06.854327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:06.854679 (+ 352us) server_negotiation.cc:407] Connection header received
0924 10:08:06.854910 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:06.854916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:06.855011 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:06.855151 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:06.855984 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60510: BlockingRecv error: recv got EOF from 127.0.0.1:60510 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260924 10:08:06.856500 6155 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:06.858227 6155 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:06.859632 6155 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:08:06.874563 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:06.861169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.6.2.193:42241)
0924 10:08:06.861381 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:06.861387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:06.861553 (+ 166us) server_negotiation.cc:407] Connection header received
0924 10:08:06.861824 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:06.861829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:06.861903 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:06.862010 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:06.862947 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.863741 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.864926 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.865400 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.865540 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:06.869326 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:06.869331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:06.869335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:06.869376 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:06.872039 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:06.872686 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:06.872693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:06.872697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:06.872826 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:06.873197 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:06.873203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:06.873207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:06.873480 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:06.873745 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:06.874035 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 10:08:06.874236 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 10:08:06.879535 12266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:06.877570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.6.2.193:42241)
0924 10:08:06.877800 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:06.877808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:06.878074 (+ 266us) server_negotiation.cc:407] Connection header received
0924 10:08:06.878281 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:06.878287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:06.878374 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:06.878483 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:06.879277 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60536: BlockingRecv error: recv got EOF from 127.0.0.1:60536 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 10:08:06.879741 6155 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790244437048964-6155-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:06.881419 6155 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:06.882910 6155 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:08:06 dist-test-slave-s3n3 krb5kdc[11757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:08:06.898058 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:06.884306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.6.2.193:42241)
0924 10:08:06.884527 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:06.884533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:06.884735 (+ 202us) server_negotiation.cc:407] Connection header received
0924 10:08:06.884915 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:06.884919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:06.884980 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:06.885074 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:06.886020 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.886813 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.888240 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.888691 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.888794 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:06.892518 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:06.892524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:06.892527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:06.892564 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:06.895271 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:06.896034 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:06.896038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:06.896041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:06.896227 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:06.896677 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:06.896680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:06.896682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:06.896943 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:06.897172 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:06.897477 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 10:08:06.897687 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 10:08:06.899924 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 11863
I20260924 10:08:06.913633 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:06.900816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.6.2.193:42241)
0924 10:08:06.900972 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:06.900976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:06.901289 (+ 313us) server_negotiation.cc:407] Connection header received
0924 10:08:06.901626 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:06.901630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:06.901685 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:06.901771 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:06.902763 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.903570 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.904789 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:06.905264 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:06.905362 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:06.907727 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:06.907732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:06.907736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:06.907769 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:06.910829 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:06.911621 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:06.911625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:06.911627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:06.911756 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:06.912095 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:06.912099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:06.912101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:06.912354 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:06.912564 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:06.913165 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 10:08:06.913349 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:08:06.914330 11986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:07.001988 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 11863
I20260924 10:08:07.019613 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 11997
Sep 24 10:08:07 dist-test-slave-s3n3 krb5kdc[11757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:08:07.037837 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:07.020849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.6.2.194:43689)
0924 10:08:07.021391 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:07.021398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:07.021423 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:08:07.021489 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:07.021495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:07.021759 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:07.021969 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:07.022951 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:07.024656 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:07.026304 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:07.026996 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:07.027223 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:07.031537 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:07.031550 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:07.031567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:07.031618 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:07.034782 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:07.035770 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:07.035781 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:07.035786 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:07.035947 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:07.036309 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:07.036313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:07.036315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:07.036727 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:07.037038 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:07.037187 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 10:08:07.037486 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":272,"threads_started":1}
I20260924 10:08:07.039268 12120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:07.124485 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 11997
I20260924 10:08:07.140713 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 12130
Sep 24 10:08:07 dist-test-slave-s3n3 krb5kdc[11757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:08:07.166066 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:07.141852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.6.2.195:40271)
0924 10:08:07.142507 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:07.142515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:07.142546 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:08:07.142611 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:07.142618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:07.142822 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:07.143029 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:07.144087 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:07.145084 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:07.146329 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:07.146899 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:07.147044 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:07.159313 (+ 12269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:07.159330 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:07.159349 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:07.159393 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:07.162473 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:07.163661 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:07.163670 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:07.163672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:07.163820 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:07.164271 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:07.164276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:07.164278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:07.164694 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:07.164954 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:07.165467 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 10:08:07.165778 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":368,"threads_started":1}
I20260924 10:08:07.167487 12253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:07.271608 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 12130
I20260924 10:08:07.293387 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 11770
I20260924 10:08:07.294761 11832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:07.402377 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 11770
2026-09-24T10:08:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4110 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:08:07 dist-test-slave-s3n3 krb5kdc[12286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:08:07 dist-test-slave-s3n3 krb5kdc[12286](info): set up 2 sockets
Sep 24 10:08:07 dist-test-slave-s3n3 krb5kdc[12286](info): commencing operation
krb5kdc: starting...
W20260924 10:08:08.547994 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:08:08 dist-test-slave-s3n3 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:08:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:08:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.254.keytab.
Entry for principal oryx/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.254.keytab.
I20260924 10:08:08.771661 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:41877
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.254.keytab
--principal=oryx/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:41877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:09.112684 12299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:09.113051 12299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:09.113219 12299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:09.121309 12299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:09.121502 12299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:09.121618 12299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:09.121713 12299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:09.121797 12299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:08:09.132787 12299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:41877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.254.keytab
--principal=oryx/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:41877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:09.134186 12299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:09.135591 12299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:09.147640 12307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:09.147727 12305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:09.147678 12304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:09.149032 12299 server_base.cc:1034] running on GCE node
I20260924 10:08:09.149837 12299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:09.151494 12299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:09.152742 12299 hybrid_clock.cc:648] HybridClock initialized: now 1790244489152697 us; error 42 us; skew 500 ppm
I20260924 10:08:09.153107 12299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:09 dist-test-slave-s3n3 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244489, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:09.156422 12299 init.cc:377] Logged in from keytab as oryx/127.6.2.254@KRBTEST.COM (short username oryx)
I20260924 10:08:09.158002 12299 webserver.cc:460] Webserver started at http://127.6.2.254:37909/ using document root <none> and password file <none>
I20260924 10:08:09.158674 12299 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:09.158772 12299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:09.159025 12299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:09.162117 12299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "1930212695dd4fe8b9c68e351946640d"
format_stamp: "Formatted at 2026-09-24 10:08:09 on dist-test-slave-s3n3"
server_key: "8ad307c46708fa1ca9974d4723fac525"
server_key_iv: "f72243fa309e2cf4e1da988c9a2e5a03"
server_key_version: "encryptionkey@0"
I20260924 10:08:09.162843 12299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "1930212695dd4fe8b9c68e351946640d"
format_stamp: "Formatted at 2026-09-24 10:08:09 on dist-test-slave-s3n3"
server_key: "8ad307c46708fa1ca9974d4723fac525"
server_key_iv: "f72243fa309e2cf4e1da988c9a2e5a03"
server_key_version: "encryptionkey@0"
I20260924 10:08:09.168357 12299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 10:08:09.171975 12314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:09.173377 12299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:08:09.173563 12299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "1930212695dd4fe8b9c68e351946640d"
format_stamp: "Formatted at 2026-09-24 10:08:09 on dist-test-slave-s3n3"
server_key: "8ad307c46708fa1ca9974d4723fac525"
server_key_iv: "f72243fa309e2cf4e1da988c9a2e5a03"
server_key_version: "encryptionkey@0"
I20260924 10:08:09.173760 12299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:09.194355 12299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:09.198824 12299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:09.199379 12299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:09.221254 12299 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:41877
I20260924 10:08:09.221294 12365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:41877 every 8 connection(s)
I20260924 10:08:09.222693 12299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:08:09.224538 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 12299
I20260924 10:08:09.224756 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:08:09.225462 6155 external_mini_cluster.cc:1428] Setting key a0f92dee4d22d03683bd676d09d0ef0f
I20260924 10:08:09.227772 12366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:09.241811 12366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d: Bootstrap starting.
Sep 24 10:08:09 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:09.245726 12366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:09.246938 12366 log.cc:824] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:09.250427 12366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d: No bootstrap required, opened a new log
I20260924 10:08:09.251333 12369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:09.228157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.6.2.254:41877)
0924 10:08:09.228934 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:09.228949 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:09.229000 (+ 51us) server_negotiation.cc:407] Connection header received
0924 10:08:09.231013 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:09.231040 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:09.231631 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:09.232099 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:09.233517 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:09.235637 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:09.237488 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:09.238579 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:09.238784 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:09.243407 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:09.243415 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:09.243432 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:09.243475 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:09.246721 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:09.247659 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:09.247671 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:09.247681 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:09.247914 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:09.248342 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:09.248350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:09.248356 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:09.248929 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:09.249149 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:09.249733 (+ 584us) server_negotiation.cc:299] Negotiation successful
0924 10:08:09.250159 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":329,"threads_started":1}
I20260924 10:08:09.258808 12366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } }
I20260924 10:08:09.259302 12366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:09.259424 12366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1930212695dd4fe8b9c68e351946640d, State: Initialized, Role: FOLLOWER
I20260924 10:08:09.260144 12366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } }
I20260924 10:08:09.260458 12366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:08:09.260623 12366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:08:09.260808 12366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:09.262974 12366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } }
I20260924 10:08:09.263538 12366 leader_election.cc:302] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1930212695dd4fe8b9c68e351946640d; no voters:
I20260924 10:08:09.264142 12366 leader_election.cc:288] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:08:09.264343 12371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:09.265162 12371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [term 1 LEADER]: Becoming Leader. State: Replica: 1930212695dd4fe8b9c68e351946640d, State: Running, Role: LEADER
I20260924 10:08:09.265664 12371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } }
I20260924 10:08:09.266392 12366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:08:09.268647 12373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1930212695dd4fe8b9c68e351946640d. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } }
I20260924 10:08:09.268560 12372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } }
I20260924 10:08:09.268971 12373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:09.268970 12372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:09.270037 12377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:08:09.276980 12377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:08:09.287784 12377 catalog_manager.cc:1347] Generated new cluster ID: 3579f76975e44464926a9da06da8f9a6
I20260924 10:08:09.287922 12377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:08:09.297420 12377 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:08:09.298830 12377 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:08:09.305727 12377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d: Generated new TSK 0
I20260924 10:08:09.306631 12377 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab.
Entry for principal oryx/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab.
I20260924 10:08:09.371003 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41877
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab
--principal=oryx/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:09.705819 12392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:09.706161 12392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:09.706331 12392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:09.714080 12392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:09.714253 12392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:09.714445 12392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:08:09.724700 12392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab
--principal=oryx/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:09.726075 12392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:09.727494 12392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:09.741042 12400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:09.741003 12397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:09.741098 12398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:09.742164 12392 server_base.cc:1034] running on GCE node
I20260924 10:08:09.742812 12392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:09.743981 12392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:09.745231 12392 hybrid_clock.cc:648] HybridClock initialized: now 1790244489745176 us; error 48 us; skew 500 ppm
I20260924 10:08:09.745639 12392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:09 dist-test-slave-s3n3 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244489, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:09.748932 12392 init.cc:377] Logged in from keytab as oryx/127.6.2.193@KRBTEST.COM (short username oryx)
I20260924 10:08:09.750478 12392 webserver.cc:460] Webserver started at http://127.6.2.193:43313/ using document root <none> and password file <none>
I20260924 10:08:09.751497 12392 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:09.751675 12392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:09.752127 12392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:09.755409 12392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "cbe69c95bbae4afaaf1af62cbb2e918f"
format_stamp: "Formatted at 2026-09-24 10:08:09 on dist-test-slave-s3n3"
server_key: "d6aee6c22a2de55493c2eecab280bb00"
server_key_iv: "a590459aaf0e6b6a2b69606c44f66b51"
server_key_version: "encryptionkey@0"
I20260924 10:08:09.756278 12392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "cbe69c95bbae4afaaf1af62cbb2e918f"
format_stamp: "Formatted at 2026-09-24 10:08:09 on dist-test-slave-s3n3"
server_key: "d6aee6c22a2de55493c2eecab280bb00"
server_key_iv: "a590459aaf0e6b6a2b69606c44f66b51"
server_key_version: "encryptionkey@0"
I20260924 10:08:09.762043 12392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 10:08:09.766059 12407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:09.767781 12392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:08:09.767957 12392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "cbe69c95bbae4afaaf1af62cbb2e918f"
format_stamp: "Formatted at 2026-09-24 10:08:09 on dist-test-slave-s3n3"
server_key: "d6aee6c22a2de55493c2eecab280bb00"
server_key_iv: "a590459aaf0e6b6a2b69606c44f66b51"
server_key_version: "encryptionkey@0"
I20260924 10:08:09.768190 12392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:09.784026 12392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:09.788282 12392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:09.788597 12392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:09.789539 12392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:09.790905 12392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:09.791013 12392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:09.791177 12392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:09.791257 12392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:09.827455 12392 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:45265
I20260924 10:08:09.827500 12519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:45265 every 8 connection(s)
I20260924 10:08:09.828737 12392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:08:09.835567 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 12392
I20260924 10:08:09.835840 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:08:09.836557 6155 external_mini_cluster.cc:1428] Setting key fc84cce80007cf7eb9e8c4e098aa912a
Sep 24 10:08:09 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244489, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:09.851872 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:09.831620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:43797 (local address 127.6.2.254:41877)
0924 10:08:09.832423 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:09.832428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:09.832984 (+ 556us) server_negotiation.cc:407] Connection header received
0924 10:08:09.834793 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:09.834802 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:09.834901 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:09.835086 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:09.836902 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:09.837923 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:09.839895 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:09.840759 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:09.840925 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:09.844803 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:09.844813 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:09.844822 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:09.844873 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:09.847901 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:09.848958 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:09.848962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:09.848964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:09.849094 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:09.849807 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:09.849813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:09.849816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:09.850259 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:09.850376 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:09.851416 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0924 10:08:09.851613 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":434,"threads_started":1}
I20260924 10:08:09.853544 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:09.831890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.193:43797)
0924 10:08:09.832769 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:09.832814 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:09.834509 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:09.835366 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:09.835375 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:09.836077 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:09.836708 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:09.836728 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:09.838121 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:09.838125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:09.839729 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:09.839742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:09.841010 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:09.841015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:09.841220 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:09.841960 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:09.841983 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:09.844493 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:09.848065 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:09.848079 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:09.848104 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:09.848814 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:09.849218 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:09.849226 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:09.849233 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:09.849508 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:09.850400 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:09.850412 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:09.850943 (+ 531us) client_negotiation.cc:769] Sending connection context
0924 10:08:09.851311 (+ 368us) client_negotiation.cc:240] Negotiation successful
0924 10:08:09.851954 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":370,"threads_started":1}
I20260924 10:08:09.855414 12520 heartbeater.cc:347] Connected to a master server at 127.6.2.254:41877
I20260924 10:08:09.855748 12520 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:09.856514 12520 heartbeater.cc:511] Master 127.6.2.254:41877 requested a full tablet report, sending...
I20260924 10:08:09.859107 12331 ts_manager.cc:194] Registered new tserver with Master: cbe69c95bbae4afaaf1af62cbb2e918f (127.6.2.193:45265)
WARNING: no policy specified for oryx/127.6.2.194@KRBTEST.COM; defaulting to no policy
I20260924 10:08:09.861294 12331 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:43797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab.
Entry for principal oryx/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab.
I20260924 10:08:09.891670 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41877
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab
--principal=oryx/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:10.220381 12526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:10.220762 12526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:10.220940 12526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:10.228638 12526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:10.228751 12526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:10.228919 12526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:08:10.238978 12526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab
--principal=oryx/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:10.240420 12526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:10.241803 12526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:10.255014 12534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:10.255053 12532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:10.255136 12531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:10.256107 12526 server_base.cc:1034] running on GCE node
I20260924 10:08:10.256716 12526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:10.257709 12526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:10.258934 12526 hybrid_clock.cc:648] HybridClock initialized: now 1790244490258853 us; error 67 us; skew 500 ppm
I20260924 10:08:10.259333 12526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:10 dist-test-slave-s3n3 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244490, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:10.269637 12526 init.cc:377] Logged in from keytab as oryx/127.6.2.194@KRBTEST.COM (short username oryx)
I20260924 10:08:10.271335 12526 webserver.cc:460] Webserver started at http://127.6.2.194:42237/ using document root <none> and password file <none>
I20260924 10:08:10.272198 12526 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:10.272305 12526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:10.272616 12526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:10.275687 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "356ebd95dd6a477ea2216bf6bc20e691"
format_stamp: "Formatted at 2026-09-24 10:08:10 on dist-test-slave-s3n3"
server_key: "a4fc2d3ff7d09c09c3f619a9fd8fd103"
server_key_iv: "93f01024b969c2a8c78f9dd1ea64f9ed"
server_key_version: "encryptionkey@0"
I20260924 10:08:10.276463 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "356ebd95dd6a477ea2216bf6bc20e691"
format_stamp: "Formatted at 2026-09-24 10:08:10 on dist-test-slave-s3n3"
server_key: "a4fc2d3ff7d09c09c3f619a9fd8fd103"
server_key_iv: "93f01024b969c2a8c78f9dd1ea64f9ed"
server_key_version: "encryptionkey@0"
I20260924 10:08:10.281944 12526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:08:10.285651 12541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:10.287168 12526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:08:10.287352 12526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "356ebd95dd6a477ea2216bf6bc20e691"
format_stamp: "Formatted at 2026-09-24 10:08:10 on dist-test-slave-s3n3"
server_key: "a4fc2d3ff7d09c09c3f619a9fd8fd103"
server_key_iv: "93f01024b969c2a8c78f9dd1ea64f9ed"
server_key_version: "encryptionkey@0"
I20260924 10:08:10.287606 12526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:10.306635 12526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:10.311261 12526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:10.311568 12526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:10.312561 12526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:10.313971 12526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:10.314075 12526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:10.314229 12526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:10.314301 12526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:10.347910 12526 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:39965
I20260924 10:08:10.347950 12653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:39965 every 8 connection(s)
I20260924 10:08:10.349200 12526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:08:10.353718 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 12526
I20260924 10:08:10.353904 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:08:10.354471 6155 external_mini_cluster.cc:1428] Setting key 8ed60715ddfab623e9dc3383d7a5fb29
Sep 24 10:08:10 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244490, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:10.373153 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:10.351743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:35883 (local address 127.6.2.254:41877)
0924 10:08:10.351917 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:10.351922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:10.352963 (+ 1041us) server_negotiation.cc:407] Connection header received
0924 10:08:10.355502 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:10.355506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:10.355648 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:10.355758 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:10.357638 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:10.358321 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:10.360808 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:10.361247 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:10.361349 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:10.365635 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:10.365644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:10.365652 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:10.365698 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:10.369522 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:10.370345 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:10.370353 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:10.370360 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:10.370512 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:10.370931 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:10.370938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:10.370944 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:10.371362 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:10.371544 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:10.372422 (+ 878us) server_negotiation.cc:299] Negotiation successful
0924 10:08:10.372699 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:08:10.374555 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:10.352120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.194:35883)
0924 10:08:10.352803 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:10.352840 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:10.355198 (+ 2358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:10.355954 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:10.355962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:10.356619 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:10.357401 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:10.357449 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:10.358472 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:10.358479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:10.360657 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:10.360674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:10.361490 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:10.361499 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:10.361833 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:10.362620 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:10.362660 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:10.365379 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:10.369683 (+ 4304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:10.369690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:10.369705 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:10.370204 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:10.370636 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:10.370639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:10.370642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:10.370817 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:10.371558 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:10.371565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:10.371927 (+ 362us) client_negotiation.cc:769] Sending connection context
0924 10:08:10.372275 (+ 348us) client_negotiation.cc:240] Negotiation successful
0924 10:08:10.372904 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":275,"threads_started":1}
I20260924 10:08:10.376202 12654 heartbeater.cc:347] Connected to a master server at 127.6.2.254:41877
I20260924 10:08:10.376554 12654 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:10.377511 12654 heartbeater.cc:511] Master 127.6.2.254:41877 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:08:10.379324 12331 ts_manager.cc:194] Registered new tserver with Master: 356ebd95dd6a477ea2216bf6bc20e691 (127.6.2.194:39965)
I20260924 10:08:10.380177 12331 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:35883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab.
Entry for principal oryx/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab.
I20260924 10:08:10.408752 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41877
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab
--principal=oryx/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:10.740692 12659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:10.740985 12659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:10.741156 12659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:10.748878 12659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:10.748996 12659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:10.749181 12659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:08:10.759155 12659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab
--principal=oryx/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:10.760568 12659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:10.761909 12659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:10.775760 12664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:10.775898 12667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:10.775753 12665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:10.777347 12659 server_base.cc:1034] running on GCE node
I20260924 10:08:10.778038 12659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:10.779400 12659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:10.780653 12659 hybrid_clock.cc:648] HybridClock initialized: now 1790244490780548 us; error 119 us; skew 500 ppm
I20260924 10:08:10.781013 12659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:10 dist-test-slave-s3n3 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244490, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:10.784360 12659 init.cc:377] Logged in from keytab as oryx/127.6.2.195@KRBTEST.COM (short username oryx)
I20260924 10:08:10.786087 12659 webserver.cc:460] Webserver started at http://127.6.2.195:46637/ using document root <none> and password file <none>
I20260924 10:08:10.786819 12659 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:10.786984 12659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:10.787395 12659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:10.790442 12659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "7258ad39bc4a4b29ab53cd2483c8c591"
format_stamp: "Formatted at 2026-09-24 10:08:10 on dist-test-slave-s3n3"
server_key: "6fcd123eaacb67e33afdfac00af72f8e"
server_key_iv: "5a830690da46b959fb9cecafa679404e"
server_key_version: "encryptionkey@0"
I20260924 10:08:10.791311 12659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "7258ad39bc4a4b29ab53cd2483c8c591"
format_stamp: "Formatted at 2026-09-24 10:08:10 on dist-test-slave-s3n3"
server_key: "6fcd123eaacb67e33afdfac00af72f8e"
server_key_iv: "5a830690da46b959fb9cecafa679404e"
server_key_version: "encryptionkey@0"
I20260924 10:08:10.796638 12659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 10:08:10.800323 12674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:10.801762 12659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:08:10.801970 12659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "7258ad39bc4a4b29ab53cd2483c8c591"
format_stamp: "Formatted at 2026-09-24 10:08:10 on dist-test-slave-s3n3"
server_key: "6fcd123eaacb67e33afdfac00af72f8e"
server_key_iv: "5a830690da46b959fb9cecafa679404e"
server_key_version: "encryptionkey@0"
I20260924 10:08:10.802189 12659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:10.829910 12659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:10.834693 12659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:10.835049 12659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:10.836166 12659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:10.837879 12659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:10.837953 12659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:10.838248 12659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:10.838292 12659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:10.864966 12520 heartbeater.cc:503] Master 127.6.2.254:41877 was elected leader, sending a full tablet report...
I20260924 10:08:10.874492 12659 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:37877
I20260924 10:08:10.874578 12786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:37877 every 8 connection(s)
I20260924 10:08:10.875839 12659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:08:10.881631 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 12659
I20260924 10:08:10.881830 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:08:10.882396 6155 external_mini_cluster.cc:1428] Setting key 45e7381480e14dc910d7d0ea20dd05a4
Sep 24 10:08:10 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244490, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:10.897331 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:10.878493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:58937 (local address 127.6.2.254:41877)
0924 10:08:10.878984 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:10.878989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:10.879971 (+ 982us) server_negotiation.cc:407] Connection header received
0924 10:08:10.881870 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:10.881876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:10.881991 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:10.882125 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:10.884176 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:10.885120 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:10.886765 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:10.887279 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:10.887380 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:10.890957 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:10.890962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:10.890965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:10.891001 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:10.893660 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:10.894454 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:10.894458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:10.894461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:10.894653 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:10.895260 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:10.895264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:10.895266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:10.895538 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:10.895750 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:10.896653 (+ 903us) server_negotiation.cc:299] Negotiation successful
0924 10:08:10.896899 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":272,"threads_started":1}
I20260924 10:08:10.898478 12790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:10.878987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.195:58937)
0924 10:08:10.879786 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:10.879832 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:10.881552 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:10.882374 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:10.882385 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:10.883202 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:10.883972 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:10.883993 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:10.885328 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:10.885333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:10.886628 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:10.886640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:10.887472 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:10.887477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:10.887719 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:10.888428 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:10.888452 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:10.890732 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:10.893812 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:10.893819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:10.893835 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:10.894315 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:10.894794 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:10.894798 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:10.894801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:10.895003 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:10.895726 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:10.895733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:10.896194 (+ 461us) client_negotiation.cc:769] Sending connection context
0924 10:08:10.896488 (+ 294us) client_negotiation.cc:240] Negotiation successful
0924 10:08:10.897110 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":303,"threads_started":1}
I20260924 10:08:10.899893 12787 heartbeater.cc:347] Connected to a master server at 127.6.2.254:41877
I20260924 10:08:10.900166 12787 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:10.900868 12787 heartbeater.cc:511] Master 127.6.2.254:41877 requested a full tablet report, sending...
I20260924 10:08:10.902259 12331 ts_manager.cc:194] Registered new tserver with Master: 7258ad39bc4a4b29ab53cd2483c8c591 (127.6.2.195:37877)
I20260924 10:08:10.903141 12331 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:58937
I20260924 10:08:10.910688 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.2.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.2.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.2.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.253.keytab.
Entry for principal oryx/127.6.2.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.253.keytab.
I20260924 10:08:10.961164 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.253:38781
--webserver_interface=127.6.2.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.253.keytab
--principal=oryx/127.6.2.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:41877,127.6.2.253:38781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:11.311197 12793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:11.311621 12793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:11.311777 12793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:11.320020 12793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:11.320182 12793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:11.320299 12793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:11.320396 12793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:11.320488 12793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:08:11.330782 12793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:41877,127.6.2.253:38781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.253.keytab
--principal=oryx/127.6.2.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.253:38781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.2.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
W20260924 10:08:11.331244 12793 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.2.254:41877,127.6.2.253:38781'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 10:08:11.332338 12793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:11.333678 12793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:11.345738 12798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:11.345752 12799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:11.345839 12801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:11.346678 12793 server_base.cc:1034] running on GCE node
I20260924 10:08:11.347724 12793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:11.349599 12793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:11.350867 12793 hybrid_clock.cc:648] HybridClock initialized: now 1790244491350762 us; error 91 us; skew 500 ppm
I20260924 10:08:11.351286 12793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:11 dist-test-slave-s3n3 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244491, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:11.355331 12793 init.cc:377] Logged in from keytab as oryx/127.6.2.253@KRBTEST.COM (short username oryx)
I20260924 10:08:11.356863 12793 webserver.cc:460] Webserver started at http://127.6.2.253:40175/ using document root <none> and password file <none>
I20260924 10:08:11.357651 12793 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:11.357746 12793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:11.358050 12793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:11.361119 12793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/data/instance:
uuid: "9dd49fc19ab84da6aa61722da0b2afc0"
format_stamp: "Formatted at 2026-09-24 10:08:11 on dist-test-slave-s3n3"
server_key: "47dd2268b48cc599c29dd57fd6c14860"
server_key_iv: "645a883dd86e8fd2eee2909bd0949b61"
server_key_version: "encryptionkey@0"
I20260924 10:08:11.361907 12793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/wal/instance:
uuid: "9dd49fc19ab84da6aa61722da0b2afc0"
format_stamp: "Formatted at 2026-09-24 10:08:11 on dist-test-slave-s3n3"
server_key: "47dd2268b48cc599c29dd57fd6c14860"
server_key_iv: "645a883dd86e8fd2eee2909bd0949b61"
server_key_version: "encryptionkey@0"
I20260924 10:08:11.367645 12793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 10:08:11.371443 12808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:11.373476 12793 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:08:11.373692 12793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/wal
uuid: "9dd49fc19ab84da6aa61722da0b2afc0"
format_stamp: "Formatted at 2026-09-24 10:08:11 on dist-test-slave-s3n3"
server_key: "47dd2268b48cc599c29dd57fd6c14860"
server_key_iv: "645a883dd86e8fd2eee2909bd0949b61"
server_key_version: "encryptionkey@0"
I20260924 10:08:11.373947 12793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:11.382798 12654 heartbeater.cc:503] Master 127.6.2.254:41877 was elected leader, sending a full tablet report...
I20260924 10:08:11.395673 12793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:11.400502 12793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:11.400756 12793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:11.422405 12793 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.253:38781
I20260924 10:08:11.422426 12859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.253:38781 every 8 connection(s)
I20260924 10:08:11.424001 12793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/data/info.pb
I20260924 10:08:11.424719 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 12793
I20260924 10:08:11.428964 12860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:11.437683 12860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:08:11 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.2.253@KRBTEST.COM
Sep 24 10:08:11 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244491, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.253@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:11.451553 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:11.427243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.6.2.253:38781)
0924 10:08:11.428008 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:11.428023 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:11.428073 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:08:11.430301 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:11.430328 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:11.430797 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:11.431256 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:11.432690 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.434883 (+ 2193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.436360 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.437410 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.437621 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:11.442672 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:11.442684 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:11.442699 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:11.442757 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:11.446526 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.447535 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.447541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.447548 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.447831 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.448269 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.448275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.448278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.449024 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:11.449334 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:11.449758 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 10:08:11.450198 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":321,"threads_started":1}
I20260924 10:08:11.457499 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:11.440305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:60026)
0924 10:08:11.441259 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:11.441315 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:11.441516 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:11.442289 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:11.442293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:11.443193 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:11.443551 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:11.443561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.444829 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.444834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:11.446651 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:11.446671 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.447790 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.447796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:11.448001 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:11.448685 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:11.448705 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:11.451325 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:11.455166 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:11.455174 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:11.455179 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:11.455553 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:11.455921 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:11.455924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:11.455927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:11.456037 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:11.456670 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:11.456676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:11.456812 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 10:08:11.456954 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 10:08:11.457192 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":421,"threads_started":1}
I20260924 10:08:11.457638 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:11.440412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60026 (local address 127.6.2.254:41877)
0924 10:08:11.441898 (+ 1486us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:11.441905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:11.441926 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:08:11.442005 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:11.442010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:11.442093 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:11.442232 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:11.443765 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.444686 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.446821 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.447563 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.447667 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:11.451517 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:11.451522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:11.451525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:11.451561 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:11.454965 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.455692 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.455696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.455698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.455809 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.456166 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.456171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.456174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.456521 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:11.456644 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:11.457130 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 10:08:11.457317 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1358,"thread_start_us":1265,"threads_started":1}
I20260924 10:08:11.459153 12860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:08:11 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244491, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.253@KRBTEST.COM for oryx/127.6.2.253@KRBTEST.COM
I20260924 10:08:11.472223 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:11.460013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:38781 (local address 127.0.0.1:56668)
0924 10:08:11.460249 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:11.460269 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:11.460406 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:11.460705 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:11.460709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:11.461051 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:11.461181 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:11.461189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.462133 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.462140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:11.462996 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:11.463006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.463699 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.463704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:11.463890 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:11.464695 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:11.464712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:11.466872 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:11.469711 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:11.469715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:11.469718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:11.470156 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:11.470525 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:11.470532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:11.470538 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:11.470675 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:11.471224 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:11.471228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:11.471383 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 10:08:11.471532 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 10:08:11.471825 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 10:08:11.472294 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:11.460024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.6.2.253:38781)
0924 10:08:11.460236 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:11.460246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:11.460344 (+ 98us) server_negotiation.cc:407] Connection header received
0924 10:08:11.460530 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:11.460533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:11.460589 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:11.460691 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:11.461308 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.461998 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.463175 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.463558 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.463688 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:11.467053 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:11.467113 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:11.467120 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:11.467170 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:11.469555 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.470267 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.470274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.470279 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.470413 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.470778 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.470785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.470789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.471046 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:11.471266 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:11.471610 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 10:08:11.471972 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":85}
I20260924 10:08:11.478394 12860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: Bootstrap starting.
I20260924 10:08:11.482138 12860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:11.483465 12860 log.cc:824] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:11.486390 12860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: No bootstrap required, opened a new log
I20260924 10:08:11.494515 12860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } }
I20260924 10:08:11.494980 12860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:11.495152 12860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd49fc19ab84da6aa61722da0b2afc0, State: Initialized, Role: FOLLOWER
I20260924 10:08:11.495718 12860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } }
I20260924 10:08:11.496595 12867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } } }
I20260924 10:08:11.496876 12867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:08:11.497649 12860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 10:08:11.507702 12878 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:08:11.507866 12878 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:08:11.509995 12793 master_runner.cc:428] Detected that this master 9dd49fc19ab84da6aa61722da0b2afc0 is joining an existing cluster
I20260924 10:08:11.510114 12793 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.2.253:38781
WARNING: no policy specified for oryx/127.6.2.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.2.252@KRBTEST.COM" created.
I20260924 10:08:11.543576 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:11.523632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.6.2.253:38781)
0924 10:08:11.523855 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:11.523862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:11.527187 (+ 3325us) server_negotiation.cc:407] Connection header received
0924 10:08:11.527372 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:11.527376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:11.527436 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:11.527502 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:11.529501 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.530588 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.532948 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.533503 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.533646 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:11.535972 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:11.535978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:11.535983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:11.536030 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:11.540789 (+ 4759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.541444 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.541449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.541453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.541591 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.541959 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.541963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.541966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.542265 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:11.542440 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:11.542837 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 10:08:11.543232 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":153,"server-negotiator.queue_time_us":55}
I20260924 10:08:11.543612 12886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:11.523661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:38781 (local address 127.0.0.1:56684)
0924 10:08:11.527103 (+ 3442us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:11.527120 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:11.527258 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:11.527706 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:11.527711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:11.528668 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:11.529350 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:11.529364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.530796 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.530802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:11.532833 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:11.532845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.533685 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.533692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:11.533866 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:11.534554 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:11.534573 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:11.535771 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:11.540923 (+ 5152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:11.540928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:11.540932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:11.541313 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:11.541704 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:11.541710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:11.541713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:11.541845 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:11.542378 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:11.542382 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:11.542600 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 10:08:11.542737 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 10:08:11.543065 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3303,"mutex_wait_us":198,"spinlock_wait_cycles":30592,"thread_start_us":3138,"threads_started":1}
I20260924 10:08:11.547255 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:11.517692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60034 (local address 127.6.2.254:41877)
0924 10:08:11.517925 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:11.517934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:11.526778 (+ 8844us) server_negotiation.cc:407] Connection header received
0924 10:08:11.527320 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:11.527330 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:11.527447 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:11.527513 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:11.529517 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.530588 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.532897 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.533727 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.533877 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:11.536403 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:11.536410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:11.536415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:11.536455 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:11.540155 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.544794 (+ 4639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.544801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.544804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.544975 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.545447 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.545452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.545456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.545820 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:11.545959 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:11.546627 (+ 668us) server_negotiation.cc:299] Negotiation successful
0924 10:08:11.546820 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 10:08:11.547284 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:11.517702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:60034)
0924 10:08:11.526618 (+ 8916us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:11.526643 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:11.527152 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:11.527706 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:11.527711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:11.528668 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:11.529361 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:11.529370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.530758 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.530768 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:11.532706 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:11.532723 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.534147 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.534154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:11.534361 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:11.534989 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:11.535012 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:11.536170 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:11.544129 (+ 7959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:11.544135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:11.544151 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:11.544630 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:11.545130 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:11.545135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:11.545150 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:11.545312 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:11.546038 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:11.546044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:11.546273 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 10:08:11.546444 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 10:08:11.546760 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8762,"mutex_wait_us":170,"thread_start_us":8597,"threads_started":1}
Sep 24 10:08:11 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244489, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.254@KRBTEST.COM for oryx/127.6.2.253@KRBTEST.COM
I20260924 10:08:11.568780 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:11.552782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.6.2.253:38781)
0924 10:08:11.553103 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:11.553109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:11.553509 (+ 400us) server_negotiation.cc:407] Connection header received
0924 10:08:11.553835 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:11.553841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:11.553950 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:11.554066 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:11.555251 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.555974 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.557317 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.557890 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.558042 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:11.562171 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:11.562177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:11.562188 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:11.562221 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:11.565339 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.566322 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.566330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.566335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.566468 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.567141 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.567145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.567147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.567413 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:11.567573 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:11.568140 (+ 567us) server_negotiation.cc:299] Negotiation successful
0924 10:08:11.568374 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260924 10:08:11.568871 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:11.552768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:38781 (local address 127.0.0.1:56700)
0924 10:08:11.553382 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:11.553425 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:11.553647 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:11.554106 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:11.554114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:11.554797 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:11.555004 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:11.555014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.556113 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.556121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:11.557186 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:11.557197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.558067 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.558077 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:11.558280 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:11.559373 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:11.559436 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:11.561964 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:11.565487 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:11.565502 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:11.565510 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:11.566192 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:11.566704 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:11.566711 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:11.566714 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:11.566947 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:11.567578 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:11.567589 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:11.567849 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 10:08:11.568056 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 10:08:11.568450 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":343,"threads_started":1}
I20260924 10:08:11.571338 12331 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.2.253:38781: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0"
member_type: NON_VOTER
last_known_addr {
host: "127.6.2.253"
port: 38781
}
attrs {
promote: true
}
}
dest_uuid: "1930212695dd4fe8b9c68e351946640d"
cas_config_opid_index: -1
I20260924 10:08:11.572644 12331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: NON_VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.2.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.252.keytab.
Entry for principal oryx/127.6.2.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.252.keytab.
I20260924 10:08:11.577276 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.252:45041
--webserver_interface=127.6.2.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.252.keytab
--principal=oryx/127.6.2.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:41877,127.6.2.253:38781,127.6.2.252:45041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:08:11.588326 12835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:11.588282 12891 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9dd49fc19ab84da6aa61722da0b2afc0 (127.6.2.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: NON_VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: true } } }
I20260924 10:08:11.591683 12835 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 FOLLOWER]: Refusing update from remote peer 1930212695dd4fe8b9c68e351946640d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 10:08:11.593096 12893 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: NON_VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:11.595177 12891 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.2.253:38781
I20260924 10:08:11.595901 12793 master.cc:537] Master@127.6.2.253:38781 shutting down...
I20260924 10:08:11.596717 12890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: NON_VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: true } } }
I20260924 10:08:11.597039 12890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:11.636834 12835 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9dd49fc19ab84da6aa61722da0b2afc0 (127.6.2.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: NON_VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: true } } }
I20260924 10:08:11.656260 12901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: NON_VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: true } } }
I20260924 10:08:11.656812 12901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: This master's current role is: LEARNER
I20260924 10:08:11.656889 12897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: NON_VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: true } } }
I20260924 10:08:11.657401 12897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: This master's current role is: LEARNER
I20260924 10:08:11.657402 12867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1930212695dd4fe8b9c68e351946640d. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: NON_VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: true } } }
I20260924 10:08:11.658316 12867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: This master's current role is: LEARNER
I20260924 10:08:11.660437 12793 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 LEARNER]: Raft consensus shutting down.
I20260924 10:08:11.660897 12793 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 LEARNER]: Raft consensus is shut down!
I20260924 10:08:11.661072 12793 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: stopping tablet replica
I20260924 10:08:11.668315 12793 master.cc:552] Master@127.6.2.253:38781 shutdown complete.
I20260924 10:08:11.668438 12793 master_runner.cc:305] Clearing existing system tablet
I20260924 10:08:11.668819 12793 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:11.668929 12793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:11.671533 12793 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:08:11.673393 12903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:11.673717 12793 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:11.674024 12793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260924 10:08:11.674110 12793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/wal
uuid: "9dd49fc19ab84da6aa61722da0b2afc0"
format_stamp: "Formatted at 2026-09-24 10:08:11 on dist-test-slave-s3n3"
server_key: "47dd2268b48cc599c29dd57fd6c14860"
server_key_iv: "645a883dd86e8fd2eee2909bd0949b61"
server_key_version: "encryptionkey@0"
I20260924 10:08:11.677371 12793 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:11.683159 12793 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 10:08:11.683358 12793 log.cc:1196] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 10:08:11.683843 12793 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: Deleting consensus metadata
I20260924 10:08:11.684115 12793 master_runner.cc:315] Copying system tablet from 127.6.2.254:41877
I20260924 10:08:11.686754 12793 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.2.254:41877
I20260924 10:08:11.704545 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:11.687719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.6.2.254:41877)
0924 10:08:11.687921 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:11.687930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:11.691738 (+ 3808us) server_negotiation.cc:407] Connection header received
0924 10:08:11.692020 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:11.692025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:11.692134 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:11.692252 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:11.693262 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.694000 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.695487 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.695982 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.696065 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:11.698562 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:11.698568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:11.698571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:11.698604 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:11.701769 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.702456 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.702463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.702467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.702610 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:11.703003 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:11.703010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:11.703013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:11.703358 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:11.703517 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:11.703917 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 10:08:11.704147 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:08:11.704628 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:11.687762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:60044)
0924 10:08:11.691596 (+ 3834us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:11.691618 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:11.691876 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:11.692282 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:11.692287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:11.692796 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:11.693100 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:11.693110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.694165 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.694178 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:11.695297 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:11.695316 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:11.696300 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:11.696304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:11.696450 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:11.697192 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:11.697261 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:11.698252 (+ 991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:11.701920 (+ 3668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:11.701925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:11.701929 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:11.702326 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:11.702725 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:11.702730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:11.702734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:11.702893 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:11.703482 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:11.703487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:11.703654 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 10:08:11.703818 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 10:08:11.704167 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3691,"thread_start_us":3500,"threads_started":1}
I20260924 10:08:11.705926 12351 tablet_copy_service.cc:140] P 1930212695dd4fe8b9c68e351946640d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9dd49fc19ab84da6aa61722da0b2afc0 ({username='oryx', principal='oryx/127.6.2.253@KRBTEST.COM'} at 127.0.0.1:60044)
I20260924 10:08:11.706189 12351 tablet_copy_service.cc:161] P 1930212695dd4fe8b9c68e351946640d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9dd49fc19ab84da6aa61722da0b2afc0 at {username='oryx', principal='oryx/127.6.2.253@KRBTEST.COM'} at 127.0.0.1:60044: session id = 9dd49fc19ab84da6aa61722da0b2afc0-00000000000000000000000000000000
I20260924 10:08:11.709463 12351 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 10:08:11.711835 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 10:08:11.716826 12793 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: tablet copy: Starting download of 0 data blocks...
I20260924 10:08:11.717310 12793 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: tablet copy: Starting download of 1 WAL segments...
I20260924 10:08:11.719888 12793 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 10:08:11.723423 12351 tablet_copy_service.cc:342] P 1930212695dd4fe8b9c68e351946640d: Request end of tablet copy session 9dd49fc19ab84da6aa61722da0b2afc0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.2.253@KRBTEST.COM'} at 127.0.0.1:60044
I20260924 10:08:11.723637 12351 tablet_copy_service.cc:433] P 1930212695dd4fe8b9c68e351946640d: ending tablet copy session 9dd49fc19ab84da6aa61722da0b2afc0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9dd49fc19ab84da6aa61722da0b2afc0
I20260924 10:08:11.734077 12793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:11.736922 12910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:11.748070 12911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:11.749320 12913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:11.750141 12793 server_base.cc:1034] running on GCE node
I20260924 10:08:11.750517 12793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:11.751245 12793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:11.752692 12793 hybrid_clock.cc:648] HybridClock initialized: now 1790244491752614 us; error 38 us; skew 500 ppm
I20260924 10:08:11.752950 12793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:11 dist-test-slave-s3n3 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244491, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:11.754976 12793 init.cc:377] Logged in from keytab as oryx/127.6.2.253@KRBTEST.COM (short username oryx)
I20260924 10:08:11.756261 12793 webserver.cc:460] Webserver started at http://127.6.2.253:42261/ using document root <none> and password file <none>
I20260924 10:08:11.756582 12793 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:11.756672 12793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:11.759654 12793 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:08:11.761536 12920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:11.762041 12793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260924 10:08:11.762287 12793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/wal
uuid: "9dd49fc19ab84da6aa61722da0b2afc0"
format_stamp: "Formatted at 2026-09-24 10:08:11 on dist-test-slave-s3n3"
server_key: "47dd2268b48cc599c29dd57fd6c14860"
server_key_iv: "645a883dd86e8fd2eee2909bd0949b61"
server_key_version: "encryptionkey@0"
I20260924 10:08:11.762461 12793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:11.783195 12793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:11.786958 12793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:11.831983 12793 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.253:38781
I20260924 10:08:11.832041 12971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.253:38781 every 8 connection(s)
I20260924 10:08:11.833492 12793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-1/data/info.pb
I20260924 10:08:11.838874 12972 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:08:11.843943 12972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: Bootstrap starting.
I20260924 10:08:11.861255 12972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:08:11.861853 12972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: Bootstrap complete.
I20260924 10:08:11.863348 12972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: NON_VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: true } }
I20260924 10:08:11.863607 12972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dd49fc19ab84da6aa61722da0b2afc0, State: Initialized, Role: LEARNER
I20260924 10:08:11.863965 12972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: NON_VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: true } }
I20260924 10:08:11.864845 12975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: NON_VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: true } } }
I20260924 10:08:11.865166 12972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:08:11.865263 12975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: This master's current role is: LEARNER
I20260924 10:08:11.894035 12986 catalog_manager.cc:1259] Loaded cluster ID: 3579f76975e44464926a9da06da8f9a6
I20260924 10:08:11.894165 12986 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: loading cluster ID for follower catalog manager: success
I20260924 10:08:11.897989 12986 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: acquiring CA information for follower catalog manager: success
I20260924 10:08:11.900943 12986 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:08:11.905719 12787 heartbeater.cc:503] Master 127.6.2.254:41877 was elected leader, sending a full tablet report...
I20260924 10:08:12.019348 12891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: NON_VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: true } } }
I20260924 10:08:12.019755 12891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:12.028642 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.019667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:38781 (local address 127.0.0.1:56710)
0924 10:08:12.019908 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:12.019928 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:12.020066 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:12.021506 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:12.021514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:12.021553 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:12.021790 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.021804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.023757 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.023766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.025822 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.025841 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.027656 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.027665 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.027894 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.027964 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 10:08:12.028133 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 10:08:12.028316 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260924 10:08:12.028702 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.020378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.6.2.253:38781)
0924 10:08:12.021095 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:12.021101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:12.021123 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:08:12.021195 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:12.021202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:12.021292 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:12.021430 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:12.021984 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.023564 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.026032 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.027448 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.027612 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.027915 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:12.028173 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 10:08:12.028315 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":334,"threads_started":1}
I20260924 10:08:12.030103 12975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1930212695dd4fe8b9c68e351946640d. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: NON_VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: true } } }
I20260924 10:08:12.030362 12975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: This master's current role is: LEARNER
I20260924 10:08:12.031042 12891 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d: attempting to promote NON_VOTER 9dd49fc19ab84da6aa61722da0b2afc0 to VOTER
I20260924 10:08:12.032049 12891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } }
I20260924 10:08:12.033844 12947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 LEARNER]: Refusing update from remote peer 1930212695dd4fe8b9c68e351946640d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 10:08:12.034473 12893 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 10:08:12.043500 12891 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9dd49fc19ab84da6aa61722da0b2afc0 (127.6.2.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } }
I20260924 10:08:12.044807 12947 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9dd49fc19ab84da6aa61722da0b2afc0 (127.6.2.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } }
I20260924 10:08:12.049222 12975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } }
I20260924 10:08:12.049435 12975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:08:12.051546 12890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } }
I20260924 10:08:12.051574 12893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } }
I20260924 10:08:12.051810 12890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:12.051847 12893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: This master's current role is: LEADER
W20260924 10:08:12.062008 12892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:12.062291 12892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:12.062486 12892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:12.070217 12892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:12.070333 12892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:12.070410 12892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:12.070492 12892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:12.070598 12892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:08:12.081347 12892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:41877,127.6.2.253:38781,127.6.2.252:45041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.252.keytab
--principal=oryx/127.6.2.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.252:45041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.2.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:12.082832 12892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:12.084172 12892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:12.096629 12993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:12.096568 12994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:12.097066 12996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:12.097494 12892 server_base.cc:1034] running on GCE node
I20260924 10:08:12.098233 12892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:12.099819 12892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:12.101089 12892 hybrid_clock.cc:648] HybridClock initialized: now 1790244492101015 us; error 71 us; skew 500 ppm
I20260924 10:08:12.101550 12892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:12 dist-test-slave-s3n3 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244492, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:12.104862 12892 init.cc:377] Logged in from keytab as oryx/127.6.2.252@KRBTEST.COM (short username oryx)
I20260924 10:08:12.106671 12892 webserver.cc:460] Webserver started at http://127.6.2.252:46451/ using document root <none> and password file <none>
I20260924 10:08:12.107437 12892 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:12.107594 12892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:12.107904 12892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:12.110853 12892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/data/instance:
uuid: "66ce7cf41bf942779d67d1d64eac375b"
format_stamp: "Formatted at 2026-09-24 10:08:12 on dist-test-slave-s3n3"
server_key: "15a7aa0c7916ac09d256edc29adc67d0"
server_key_iv: "97fc0893712cca9e4922c029d51aede7"
server_key_version: "encryptionkey@0"
I20260924 10:08:12.111685 12892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/wal/instance:
uuid: "66ce7cf41bf942779d67d1d64eac375b"
format_stamp: "Formatted at 2026-09-24 10:08:12 on dist-test-slave-s3n3"
server_key: "15a7aa0c7916ac09d256edc29adc67d0"
server_key_iv: "97fc0893712cca9e4922c029d51aede7"
server_key_version: "encryptionkey@0"
I20260924 10:08:12.117062 12892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 10:08:12.120815 13003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:12.122272 12892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:08:12.122512 12892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/wal
uuid: "66ce7cf41bf942779d67d1d64eac375b"
format_stamp: "Formatted at 2026-09-24 10:08:12 on dist-test-slave-s3n3"
server_key: "15a7aa0c7916ac09d256edc29adc67d0"
server_key_iv: "97fc0893712cca9e4922c029d51aede7"
server_key_version: "encryptionkey@0"
I20260924 10:08:12.122726 12892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:12.142005 12892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:12.146454 12892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:12.146763 12892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:12.168618 12892 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.252:45041
I20260924 10:08:12.168757 13054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.252:45041 every 8 connection(s)
I20260924 10:08:12.170683 12892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/data/info.pb
I20260924 10:08:12.174536 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 12892
I20260924 10:08:12.175820 13055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:12.184983 13055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:08:12 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.2.252@KRBTEST.COM
Sep 24 10:08:12 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244492, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.252@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:12.199553 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.177902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.6.2.252:45041)
0924 10:08:12.178836 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:12.178846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:12.178888 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:08:12.180932 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:12.180953 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:12.181367 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:12.181778 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:12.183489 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.184741 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.186006 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.186644 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.186844 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.191899 (+ 5055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:12.191908 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:12.191918 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:12.191957 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:12.195041 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.195818 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.195825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.195831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.195995 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.196370 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.196377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.196381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.196874 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:12.197170 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:12.197626 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 10:08:12.198050 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":481,"threads_started":1}
I20260924 10:08:12.203336 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.187785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.6.2.254:41877)
0924 10:08:12.188010 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:12.188016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:12.188460 (+ 444us) server_negotiation.cc:407] Connection header received
0924 10:08:12.188707 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:12.188715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:12.188808 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:12.188927 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:12.190024 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.190918 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.192631 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.193091 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.193226 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.196399 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:12.196404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:12.196407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:12.196438 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:12.199545 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.201371 (+ 1826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.201375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.201377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.201508 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.201910 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.201913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.201914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.202165 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:12.202324 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:12.202758 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 10:08:12.202966 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:08:12.203379 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.187536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:60054)
0924 10:08:12.188309 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:12.188356 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:12.188518 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:12.189016 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:12.189023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:12.189632 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:12.189870 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.189878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.191087 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.191094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.192476 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.192488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.193291 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.193298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.193456 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.194011 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:12.194030 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:12.196227 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:12.199736 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.199750 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.200654 (+ 904us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.201252 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.201626 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.201631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.201634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.201786 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.202310 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:12.202317 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:12.202469 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 10:08:12.202661 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 10:08:12.202964 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":497,"threads_started":1}
I20260924 10:08:12.204705 13055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:08:12 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244492, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.252@KRBTEST.COM for oryx/127.6.2.253@KRBTEST.COM
I20260924 10:08:12.216710 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.205365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:38781 (local address 127.0.0.1:56714)
0924 10:08:12.205557 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:12.205582 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:12.205729 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:12.206035 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:12.206038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:12.206340 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:12.206469 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.206477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.207646 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.207650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.208549 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.208560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.209222 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.209226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.209361 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.209882 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:12.209898 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:12.211927 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:12.214586 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.214589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.214592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.214955 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.215362 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.215365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.215368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.215489 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.215985 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:12.215989 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:12.216111 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 10:08:12.216234 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 10:08:12.216489 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 10:08:12.216786 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.205396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.6.2.253:38781)
0924 10:08:12.205602 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:12.205611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:12.205694 (+ 83us) server_negotiation.cc:407] Connection header received
0924 10:08:12.205858 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:12.205862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:12.205921 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:12.206000 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:12.206628 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.207500 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.208677 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.209081 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.209176 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.212118 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:12.212123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:12.212126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:12.212161 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:12.214450 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.215054 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.215114 (+ 60us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.215120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.215261 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.215603 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.215606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.215608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.215856 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:12.215964 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:12.216305 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 10:08:12.216491 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:08:12.217797 13055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:08:12 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244492, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.252@KRBTEST.COM for oryx/127.6.2.252@KRBTEST.COM
I20260924 10:08:12.231043 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.218524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:45041 (local address 127.0.0.1:55574)
0924 10:08:12.218778 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:12.218792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:12.218877 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:12.219241 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:12.219244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:12.219540 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:12.219678 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.219685 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.220651 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.220656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.221468 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.221478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.222226 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.222232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.222413 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.222991 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:12.223007 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:12.225027 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:12.228769 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.228773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.228776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.229118 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.229493 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.229503 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.229505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.229659 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.230266 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:12.230270 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:12.230389 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 10:08:12.230566 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 10:08:12.230765 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260924 10:08:12.231441 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.218520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.6.2.252:45041)
0924 10:08:12.218687 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:12.218691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:12.218852 (+ 161us) server_negotiation.cc:407] Connection header received
0924 10:08:12.219003 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:12.219007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:12.219118 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:12.219209 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:12.219802 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.220538 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.221620 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.222049 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.222186 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.225199 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:12.225206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:12.225211 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:12.225245 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:12.228631 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.229224 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.229229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.229232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.229360 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.229771 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.229776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.229778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.230134 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:12.230269 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:12.230664 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 10:08:12.230947 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":69}
I20260924 10:08:12.236281 13055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: Bootstrap starting.
I20260924 10:08:12.239468 13055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:12.240494 13055 log.cc:824] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:12.243341 13055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: No bootstrap required, opened a new log
I20260924 10:08:12.250548 13055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } }
I20260924 10:08:12.250942 13055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:12.251104 13055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ce7cf41bf942779d67d1d64eac375b, State: Initialized, Role: FOLLOWER
I20260924 10:08:12.251729 13055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } }
I20260924 10:08:12.252771 13061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } } }
I20260924 10:08:12.253262 13061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:08:12.254184 13055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: configured and running, proceeding with master startup.
W20260924 10:08:12.265455 13072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:08:12.265610 13072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:08:12.269637 12892 master_runner.cc:428] Detected that this master 66ce7cf41bf942779d67d1d64eac375b is joining an existing cluster
I20260924 10:08:12.269796 12892 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.2.252:45041
I20260924 10:08:12.283289 13061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:08:12.283721 13061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } }
I20260924 10:08:12.285777 13061 leader_election.cc:288] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1930212695dd4fe8b9c68e351946640d (127.6.2.254:41877), 9dd49fc19ab84da6aa61722da0b2afc0 (127.6.2.253:38781)
I20260924 10:08:12.286931 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66ce7cf41bf942779d67d1d64eac375b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" is_pre_election: true
I20260924 10:08:12.286948 12341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66ce7cf41bf942779d67d1d64eac375b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1930212695dd4fe8b9c68e351946640d" is_pre_election: true
I20260924 10:08:12.288292 13004 leader_election.cc:302] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66ce7cf41bf942779d67d1d64eac375b; no voters: 1930212695dd4fe8b9c68e351946640d, 9dd49fc19ab84da6aa61722da0b2afc0
I20260924 10:08:12.289280 13061 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 10:08:12.306033 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 12392
I20260924 10:08:12.314016 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.274643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.6.2.254:41877)
0924 10:08:12.274812 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:12.274820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:12.292214 (+ 17394us) server_negotiation.cc:407] Connection header received
0924 10:08:12.292542 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:12.292551 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:12.292658 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:12.292748 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:12.294670 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.296250 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.300698 (+ 4448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.301375 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.301471 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.305539 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:12.305546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:12.305553 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:12.305595 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:12.309827 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.311052 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.311089 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.311093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.311303 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.311869 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.311877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.311884 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.312336 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:12.312502 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:12.313264 (+ 762us) server_negotiation.cc:299] Negotiation successful
0924 10:08:12.313498 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:08:12.315240 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.274175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:60068)
0924 10:08:12.292074 (+ 17899us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:12.292103 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:12.292370 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:12.293024 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:12.293032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:12.293921 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:12.294394 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.294410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.296531 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.296539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.298919 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.298942 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.301790 (+ 2848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.301801 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.302033 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.303812 (+ 1779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:12.303839 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:12.305317 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:12.310065 (+ 4748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.310073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.310080 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.310866 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.311488 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.311496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.311502 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.311713 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.312620 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:12.312627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:12.312911 (+ 284us) client_negotiation.cc:769] Sending connection context
0924 10:08:12.314323 (+ 1412us) client_negotiation.cc:240] Negotiation successful
0924 10:08:12.314704 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17696,"thread_start_us":17229,"threads_started":1}
I20260924 10:08:12.318812 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.274176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55588 (local address 127.6.2.252:45041)
0924 10:08:12.274381 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:12.274390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:12.300503 (+ 26113us) server_negotiation.cc:407] Connection header received
0924 10:08:12.300658 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:12.300663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:12.300739 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:12.300853 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:12.302316 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.303563 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.308226 (+ 4663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.308916 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.309015 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.311687 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:12.311693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:12.311698 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:12.311732 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:12.315696 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.316626 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.316633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.316641 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.316784 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.317177 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.317184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.317187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.317574 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:12.317731 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:12.318248 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 10:08:12.318463 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:08:12.320067 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.274774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:45041 (local address 127.0.0.1:55588)
0924 10:08:12.300359 (+ 25585us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:12.300391 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:12.300518 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:12.301002 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:12.301007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:12.301756 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:12.302141 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.302153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.303772 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.303781 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.308054 (+ 4273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.308070 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.309314 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.309320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.309496 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.310201 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:12.310246 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:12.311484 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:12.315998 (+ 4514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.316003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.316007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.316495 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.316913 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.316917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.316921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.317049 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.317799 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:12.317804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:12.317973 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 10:08:12.319314 (+ 1341us) client_negotiation.cc:240] Negotiation successful
0924 10:08:12.319613 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25437,"thread_start_us":24507,"threads_started":1}
I20260924 10:08:12.325989 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.274158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.6.2.253:38781)
0924 10:08:12.274361 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:12.274367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:12.299700 (+ 25333us) server_negotiation.cc:407] Connection header received
0924 10:08:12.301083 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:12.301090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:12.301176 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:12.301272 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:12.302811 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.303850 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.305864 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.306566 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.306669 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.317652 (+ 10983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:12.317662 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:12.317669 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:12.317714 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:12.321392 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.323102 (+ 1710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.323110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.323115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.323286 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.323732 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.323739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.323745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.324204 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:12.324376 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:12.325129 (+ 753us) server_negotiation.cc:299] Negotiation successful
0924 10:08:12.325451 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:08:12.327440 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.274741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:38781 (local address 127.0.0.1:56722)
0924 10:08:12.299567 (+ 24826us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:12.299587 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:12.299752 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:12.301452 (+ 1700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:12.301457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:12.302188 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:12.302621 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.302642 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.304034 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.304044 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.305703 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.305715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.313184 (+ 7469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.313190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.313381 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.314378 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:12.314399 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:12.317186 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:12.322011 (+ 4825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.322017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.322022 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.322525 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.323441 (+ 916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.323444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.323447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.323607 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.324462 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:12.324469 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:12.324747 (+ 278us) client_negotiation.cc:769] Sending connection context
0924 10:08:12.326303 (+ 1556us) client_negotiation.cc:240] Negotiation successful
0924 10:08:12.326788 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24674,"thread_start_us":25251,"threads_started":1}
Sep 24 10:08:12 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244489, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.254@KRBTEST.COM for oryx/127.6.2.252@KRBTEST.COM
I20260924 10:08:12.338410 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.321822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55594 (local address 127.6.2.252:45041)
0924 10:08:12.321991 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:12.321996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:12.322010 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:08:12.322097 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:12.322104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:12.322191 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:12.323158 (+ 967us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:12.323329 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.324473 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.326628 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.327451 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.328523 (+ 1072us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.331466 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:12.331473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:12.331477 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:12.331511 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:12.335295 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.336147 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.336153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.336156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.336308 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.336737 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.336742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.336746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.337105 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:12.337313 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:12.337803 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 10:08:12.338040 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:08:12.338425 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.321573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:45041 (local address 127.0.0.1:55594)
0924 10:08:12.321742 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:12.321760 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:12.321855 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:12.322356 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:12.322361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:12.322862 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:12.323158 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.323174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.324649 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.324655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.326129 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.326144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.328021 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.328028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.328221 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.328845 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:12.328862 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:12.331243 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:12.335441 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.335448 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.335454 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.336010 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.336440 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.336445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.336448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.336597 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.337285 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:12.337293 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:12.337509 (+ 216us) client_negotiation.cc:769] Sending connection context
0924 10:08:12.337697 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 10:08:12.337993 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 10:08:12.340472 12331 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.2.252:45041: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b"
member_type: NON_VOTER
last_known_addr {
host: "127.6.2.252"
port: 45041
}
attrs {
promote: true
}
}
dest_uuid: "1930212695dd4fe8b9c68e351946640d"
cas_config_opid_index: 6
Sep 24 10:08:12 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.2.193@KRBTEST.COM
I20260924 10:08:12.341253 12331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: NON_VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: true } }
I20260924 10:08:12.343611 12947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 FOLLOWER]: Refusing update from remote peer 1930212695dd4fe8b9c68e351946640d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 10:08:12.343797 13030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:12.344180 12893 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 10:08:12.346469 12891 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66ce7cf41bf942779d67d1d64eac375b (127.6.2.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: NON_VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: true } } }
I20260924 10:08:12.347160 12947 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66ce7cf41bf942779d67d1d64eac375b (127.6.2.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: NON_VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: true } } }
I20260924 10:08:12.350315 12891 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.2.252:45041
I20260924 10:08:12.350663 12891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: NON_VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: true } } }
I20260924 10:08:12.350620 12890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: NON_VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: true } } }
I20260924 10:08:12.350950 12890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:12.350942 12891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:12.351028 12892 master.cc:537] Master@127.6.2.252:45041 shutting down...
I20260924 10:08:12.352640 13030 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 1 FOLLOWER]: Refusing update from remote peer 1930212695dd4fe8b9c68e351946640d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 10:08:12.353066 12975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: NON_VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: true } } }
I20260924 10:08:12.353343 12975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:08:12.353729 12891 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: NON_VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 10:08:12.363344 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.307266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.6.2.193:45265)
0924 10:08:12.327450 (+ 20184us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:12.327461 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:12.327508 (+ 47us) server_negotiation.cc:407] Connection header received
0924 10:08:12.327588 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:12.327602 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:12.329968 (+ 2366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:12.331236 (+ 1268us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:12.331933 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.333688 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.335677 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.336442 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.336562 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.342029 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:12.342048 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:12.342069 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:12.342115 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:12.358457 (+ 16342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.359546 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.359556 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.359560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.359735 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.360203 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.360208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.360212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.360652 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:12.362315 (+ 1663us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:12.362474 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 10:08:12.362775 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19990,"thread_start_us":21095,"threads_started":1}
I20260924 10:08:12.363471 12892 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:12.363929 12892 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:12.364159 12892 tablet_replica.cc:330] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: stopping tablet replica
I20260924 10:08:12.365114 12515 generic_service.cc:196] Checking for leaks (request via RPC)
W20260924 10:08:12.379927 12317 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 10:08:12.388938 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.388480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:45041 (local address 127.0.0.1:55610)
0924 10:08:12.388653 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:12.388759 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:08:12.389297 12317 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d -> Peer 66ce7cf41bf942779d67d1d64eac375b (127.6.2.252:45041): Couldn't send request to peer 66ce7cf41bf942779d67d1d64eac375b. Status: Network error: Client connection negotiation failed: client connection to 127.6.2.252:45041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 10:08:12.390267 12892 master.cc:552] Master@127.6.2.252:45041 shutdown complete.
I20260924 10:08:12.390362 12892 master_runner.cc:305] Clearing existing system tablet
I20260924 10:08:12.390779 12892 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:12.390869 12892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:12.393980 12892 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:08:12.395726 13085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:12.396067 12892 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:12.396323 12892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:08:12.396414 12892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/wal
uuid: "66ce7cf41bf942779d67d1d64eac375b"
format_stamp: "Formatted at 2026-09-24 10:08:12 on dist-test-slave-s3n3"
server_key: "15a7aa0c7916ac09d256edc29adc67d0"
server_key_iv: "97fc0893712cca9e4922c029d51aede7"
server_key_version: "encryptionkey@0"
I20260924 10:08:12.401439 12892 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:12.408480 12892 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 10:08:12.408636 12892 log.cc:1196] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 10:08:12.409104 12892 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: Deleting consensus metadata
I20260924 10:08:12.409356 12892 master_runner.cc:315] Copying system tablet from 127.6.2.254:41877
I20260924 10:08:12.411861 12892 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.2.254:41877
I20260924 10:08:12.429557 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.412691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.6.2.254:41877)
0924 10:08:12.412907 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:12.412914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:12.415616 (+ 2702us) server_negotiation.cc:407] Connection header received
0924 10:08:12.415797 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:12.415803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:12.415894 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:12.416017 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:12.417043 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.418182 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.420070 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.420698 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.420866 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.423195 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:12.423202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:12.423207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:12.423245 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:12.426661 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.427440 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.427446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.427450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.427594 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:12.428013 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:12.428018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:12.428022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:12.428362 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:12.428516 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:12.428939 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 10:08:12.429204 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 10:08:12.429603 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.412656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:60082)
0924 10:08:12.415481 (+ 2825us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:12.415505 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:12.415661 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:12.416047 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:12.416052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:12.416602 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:12.416856 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.416869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.418338 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.418344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.419904 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.419920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.420895 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.420901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.421069 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.421824 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:12.421847 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:12.422971 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:12.426808 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.426813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.426818 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.427325 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.427709 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:12.427714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:12.427718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:12.427886 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:12.428484 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:12.428489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:12.428670 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 10:08:12.428804 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 10:08:12.429154 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2692,"thread_start_us":2549,"threads_started":1}
I20260924 10:08:12.430512 12351 tablet_copy_service.cc:140] P 1930212695dd4fe8b9c68e351946640d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 66ce7cf41bf942779d67d1d64eac375b ({username='oryx', principal='oryx/127.6.2.252@KRBTEST.COM'} at 127.0.0.1:60082)
I20260924 10:08:12.430683 12351 tablet_copy_service.cc:161] P 1930212695dd4fe8b9c68e351946640d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 66ce7cf41bf942779d67d1d64eac375b at {username='oryx', principal='oryx/127.6.2.252@KRBTEST.COM'} at 127.0.0.1:60082: session id = 66ce7cf41bf942779d67d1d64eac375b-00000000000000000000000000000000
I20260924 10:08:12.434325 12351 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 10:08:12.436249 12892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 10:08:12.443264 12892 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: tablet copy: Starting download of 0 data blocks...
I20260924 10:08:12.443560 12892 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: tablet copy: Starting download of 1 WAL segments...
I20260924 10:08:12.446149 12892 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 10:08:12.450661 12351 tablet_copy_service.cc:342] P 1930212695dd4fe8b9c68e351946640d: Request end of tablet copy session 66ce7cf41bf942779d67d1d64eac375b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.2.252@KRBTEST.COM'} at 127.0.0.1:60082
I20260924 10:08:12.450809 12351 tablet_copy_service.cc:433] P 1930212695dd4fe8b9c68e351946640d: ending tablet copy session 66ce7cf41bf942779d67d1d64eac375b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 66ce7cf41bf942779d67d1d64eac375b
I20260924 10:08:12.462247 12892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:12.471949 13092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:12.471998 13093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:12.483340 12892 server_base.cc:1034] running on GCE node
W20260924 10:08:12.483230 13095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:12.484048 12892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:12.484719 12892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:12.485926 12892 hybrid_clock.cc:648] HybridClock initialized: now 1790244492485887 us; error 30 us; skew 500 ppm
I20260924 10:08:12.486173 12892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:12 dist-test-slave-s3n3 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244492, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:12.488503 12892 init.cc:377] Logged in from keytab as oryx/127.6.2.252@KRBTEST.COM (short username oryx)
I20260924 10:08:12.489846 12892 webserver.cc:460] Webserver started at http://127.6.2.252:33645/ using document root <none> and password file <none>
I20260924 10:08:12.490237 12892 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:12.490339 12892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:12.492697 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 12392
I20260924 10:08:12.503862 12892 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.004s sys 0.008s
I20260924 10:08:12.506171 13102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:12.506747 12892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:08:12.506901 12892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/wal
uuid: "66ce7cf41bf942779d67d1d64eac375b"
format_stamp: "Formatted at 2026-09-24 10:08:12 on dist-test-slave-s3n3"
server_key: "15a7aa0c7916ac09d256edc29adc67d0"
server_key_iv: "97fc0893712cca9e4922c029d51aede7"
server_key_version: "encryptionkey@0"
I20260924 10:08:12.507143 12892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:12.519783 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:45265
--local_ip_for_outbound_sockets=127.6.2.193
--tserver_master_addrs=127.6.2.254:41877
--webserver_port=43313
--webserver_interface=127.6.2.193
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab
--principal=oryx/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.2.254:41877,127.6.2.253:38781
--tserver_master_addrs=127.6.2.254:41877,127.6.2.253:38781,127.6.2.252:45041 with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:08:12.522441 12892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:12.526552 12892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:12.552451 12892 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.252:45041
I20260924 10:08:12.552503 13154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.252:45041 every 8 connection(s)
I20260924 10:08:12.553401 12892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/master-2/data/info.pb
I20260924 10:08:12.558260 13155 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:08:12.561911 13155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: Bootstrap starting.
I20260924 10:08:12.581992 13155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:08:12.582702 13155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: Bootstrap complete.
I20260924 10:08:12.584115 13155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: NON_VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: true } }
I20260924 10:08:12.584414 13155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66ce7cf41bf942779d67d1d64eac375b, State: Initialized, Role: LEARNER
I20260924 10:08:12.584661 13155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: NON_VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: true } }
I20260924 10:08:12.585636 13158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: NON_VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: true } } }
I20260924 10:08:12.585956 13155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:08:12.586113 13158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: This master's current role is: LEARNER
I20260924 10:08:12.595320 13169 catalog_manager.cc:1259] Loaded cluster ID: 3579f76975e44464926a9da06da8f9a6
I20260924 10:08:12.595444 13169 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: loading cluster ID for follower catalog manager: success
I20260924 10:08:12.597805 13169 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: acquiring CA information for follower catalog manager: success
I20260924 10:08:12.599673 13169 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:08:12.853726 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.847531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:45041 (local address 127.0.0.1:55626)
0924 10:08:12.847800 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:12.847826 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:12.847997 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:12.848787 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:12.848792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:12.848820 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:12.849016 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.849025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.850317 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.850323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.851726 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:12.851738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.852931 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.852937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:12.853103 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.853142 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 10:08:12.853265 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 10:08:12.853414 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260924 10:08:12.853747 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:12.847631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55626 (local address 127.6.2.252:45041)
0924 10:08:12.848474 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:12.848479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:12.848494 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:08:12.848550 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:12.848554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:12.848617 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:12.848756 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:12.849163 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.850160 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.851880 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:12.852743 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:12.852893 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:12.853117 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:12.853286 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 10:08:12.853424 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":603,"threads_started":1}
I20260924 10:08:12.854902 13130 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260924 10:08:12.855516 13158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1930212695dd4fe8b9c68e351946640d. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: NON_VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: true } } }
I20260924 10:08:12.855798 13158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: This master's current role is: LEARNER
W20260924 10:08:12.908036 13107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:12.908427 13107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:12.908578 13107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:12.916230 13107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:12.916342 13107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:12.916460 13107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:08:12.926213 13107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab
--principal=oryx/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:45265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=43313
--tserver_master_addrs=127.6.2.254:41877,127.6.2.253:38781,127.6.2.252:45041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:12.927551 13107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:12.928783 13107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:12.942394 13175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:12.942319 13176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:12.942358 13178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:12.943840 13107 server_base.cc:1034] running on GCE node
I20260924 10:08:12.944465 13107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:12.945415 13107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:12.946660 13107 hybrid_clock.cc:648] HybridClock initialized: now 1790244492946582 us; error 79 us; skew 500 ppm
I20260924 10:08:12.947000 13107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:12 dist-test-slave-s3n3 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244492, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:12.950352 13107 init.cc:377] Logged in from keytab as oryx/127.6.2.193@KRBTEST.COM (short username oryx)
I20260924 10:08:12.952008 13107 webserver.cc:460] Webserver started at http://127.6.2.193:43313/ using document root <none> and password file <none>
I20260924 10:08:12.952713 13107 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:12.952853 13107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:12.958464 13107 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:08:12.961392 13185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:12.962814 13107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:08:12.962991 13107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "cbe69c95bbae4afaaf1af62cbb2e918f"
format_stamp: "Formatted at 2026-09-24 10:08:09 on dist-test-slave-s3n3"
server_key: "d6aee6c22a2de55493c2eecab280bb00"
server_key_iv: "a590459aaf0e6b6a2b69606c44f66b51"
server_key_version: "encryptionkey@0"
I20260924 10:08:12.964130 13107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:12.984894 13107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:12.989530 13107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:12.989804 13107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:12.990839 13107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:12.992290 13107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:12.992409 13107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:12.992522 13107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:12.992583 13107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:13.027002 13107 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:45265
I20260924 10:08:13.027141 13297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:45265 every 8 connection(s)
I20260924 10:08:13.028438 13107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:08:13.034319 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 13107
I20260924 10:08:13.034731 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 12526
Sep 24 10:08:13 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244492, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.252@KRBTEST.COM
I20260924 10:08:13.059231 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.032834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:55649 (local address 127.6.2.252:45041)
0924 10:08:13.033098 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:13.033108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:13.033140 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:08:13.036275 (+ 3135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:13.036282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:13.036389 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:13.036474 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:13.039195 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.040563 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.043619 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.044519 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.044654 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.050754 (+ 6100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:13.050764 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:13.050772 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:13.050826 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:13.055255 (+ 4429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.056336 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.056341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.056345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.056497 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.057099 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.057107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.057110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.057484 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:13.057625 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:13.058574 (+ 949us) server_negotiation.cc:299] Negotiation successful
0924 10:08:13.058802 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 10:08:13.061477 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.032054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:45041 (local address 127.6.2.193:55649)
0924 10:08:13.032963 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:13.033011 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:13.035922 (+ 2911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:13.037105 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:13.037120 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:13.038017 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:13.038927 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:13.038953 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.040801 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.040809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:13.043417 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:13.043436 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.045074 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.045083 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:13.045478 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.046428 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:13.046467 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:13.050405 (+ 3938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:13.055419 (+ 5014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:13.055428 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:13.055453 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:13.056202 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:13.056658 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:13.056662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:13.056665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:13.056949 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:13.057721 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:13.057728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:13.058159 (+ 431us) client_negotiation.cc:769] Sending connection context
0924 10:08:13.059511 (+ 1352us) client_negotiation.cc:240] Negotiation successful
0924 10:08:13.060052 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":2089,"threads_started":1}
I20260924 10:08:13.063417 13299 heartbeater.cc:347] Connected to a master server at 127.6.2.252:45041
I20260924 10:08:13.063822 13299 heartbeater.cc:464] Registering TS with master...
Sep 24 10:08:13 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244492, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:13.064936 13299 heartbeater.cc:511] Master 127.6.2.252:45041 requested a full tablet report, sending...
Sep 24 10:08:13 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.2.194@KRBTEST.COM
Sep 24 10:08:13 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244492, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.253@KRBTEST.COM
I20260924 10:08:13.068224 13120 ts_manager.cc:194] Registered new tserver with Master: cbe69c95bbae4afaaf1af62cbb2e918f (127.6.2.193:45265)
I20260924 10:08:13.073009 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.031510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:56831 (local address 127.6.2.254:41877)
0924 10:08:13.048740 (+ 17230us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:13.048750 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:13.055293 (+ 6543us) server_negotiation.cc:407] Connection header received
0924 10:08:13.056264 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:13.056272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:13.056367 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:13.056472 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:13.057722 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.058803 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.060825 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.061424 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.061531 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.066019 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:13.066026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:13.066031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:13.066072 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:13.069780 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.070627 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.070632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.070636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.070823 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.071385 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.071390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.071395 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.071745 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:13.071924 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:13.072326 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 10:08:13.072570 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17078,"thread_start_us":16973,"threads_started":1}
I20260924 10:08:13.073035 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.032054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.193:56831)
0924 10:08:13.055168 (+ 23114us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:13.055191 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:13.056100 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:13.056669 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:13.056673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:13.057267 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:13.057565 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:13.057577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.058979 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.058984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:13.060658 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:13.060678 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.061815 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.061821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:13.062039 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.062650 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:13.062708 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:13.065758 (+ 3050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:13.069942 (+ 4184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:13.069947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:13.069952 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:13.070477 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:13.070962 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:13.070968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:13.070972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:13.071238 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:13.071884 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:13.071890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:13.072059 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 10:08:13.072214 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 10:08:13.072549 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22887,"mutex_wait_us":984,"spinlock_wait_cycles":19712,"thread_start_us":17048,"threads_started":1}
I20260924 10:08:13.074383 13298 heartbeater.cc:347] Connected to a master server at 127.6.2.254:41877
I20260924 10:08:13.074544 13298 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:13.075044 13298 heartbeater.cc:511] Master 127.6.2.254:41877 requested a full tablet report, sending...
I20260924 10:08:13.076382 13302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.031380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:51615 (local address 127.6.2.253:38781)
0924 10:08:13.048322 (+ 16942us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:13.048331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:13.050655 (+ 2324us) server_negotiation.cc:407] Connection header received
0924 10:08:13.050730 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:13.050735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:13.050832 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:13.054452 (+ 3620us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:13.056911 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.058217 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.060814 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.062253 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.062402 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.069030 (+ 6628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:13.069037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:13.069042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:13.069086 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:13.073106 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.074038 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.074044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.074048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.074250 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.074731 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.074735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.074738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.075012 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:13.075239 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:13.075790 (+ 551us) server_negotiation.cc:299] Negotiation successful
0924 10:08:13.076031 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16764,"thread_start_us":17357,"threads_started":1}
I20260924 10:08:13.076367 12331 ts_manager.cc:194] Re-registered known tserver with Master: cbe69c95bbae4afaaf1af62cbb2e918f (127.6.2.193:45265)
I20260924 10:08:13.076732 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.032770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:38781 (local address 127.6.2.193:51615)
0924 10:08:13.050383 (+ 17613us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:13.050403 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:13.050535 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:13.055400 (+ 4865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:13.055404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:13.056397 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:13.056705 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:13.056716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.058413 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.058419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:13.060607 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:13.060621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.062728 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.062738 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:13.062979 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.065674 (+ 2695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:13.065703 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:13.068764 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:13.073429 (+ 4665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:13.073434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:13.073438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:13.073895 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:13.074360 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:13.074370 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:13.074374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:13.074566 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:13.075251 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:13.075259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:13.075461 (+ 202us) client_negotiation.cc:769] Sending connection context
0924 10:08:13.075634 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 10:08:13.075976 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17470,"mutex_wait_us":82,"thread_start_us":21992,"threads_started":1}
I20260924 10:08:13.077361 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.036448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.6.2.194:39965)
0924 10:08:13.049367 (+ 12919us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:13.049376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:13.051032 (+ 1656us) server_negotiation.cc:407] Connection header received
0924 10:08:13.051168 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:13.051178 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:13.051473 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:13.051643 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:13.054962 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.056837 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.058920 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.059887 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.060025 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.068073 (+ 8048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:13.068095 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:13.068119 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:13.068188 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:13.072755 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.074015 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.074025 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.074028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.074197 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.074702 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.074707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.074711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.075206 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:13.075488 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:13.076613 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0924 10:08:13.076918 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12745,"thread_start_us":12581,"threads_started":1}
I20260924 10:08:13.077482 12331 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:56831
I20260924 10:08:13.078634 13300 heartbeater.cc:347] Connected to a master server at 127.6.2.253:38781
I20260924 10:08:13.078807 13300 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:13.078877 12649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:13.079656 13300 heartbeater.cc:511] Master 127.6.2.253:38781 requested a full tablet report, sending...
I20260924 10:08:13.081580 12937 ts_manager.cc:194] Registered new tserver with Master: cbe69c95bbae4afaaf1af62cbb2e918f (127.6.2.193:45265)
I20260924 10:08:13.166162 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 12526
I20260924 10:08:13.190439 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:39965
--local_ip_for_outbound_sockets=127.6.2.194
--tserver_master_addrs=127.6.2.254:41877
--webserver_port=42237
--webserver_interface=127.6.2.194
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab
--principal=oryx/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.2.254:41877,127.6.2.253:38781
--tserver_master_addrs=127.6.2.254:41877,127.6.2.253:38781,127.6.2.252:45041 with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:08:13.315822 12891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: NON_VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: true } } }
I20260924 10:08:13.316196 12891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:13.317255 13311 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d: attempting to promote NON_VOTER 66ce7cf41bf942779d67d1d64eac375b to VOTER
I20260924 10:08:13.317900 13311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } }
I20260924 10:08:13.319741 13130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 1 LEARNER]: Refusing update from remote peer 1930212695dd4fe8b9c68e351946640d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 10:08:13.319862 12947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 FOLLOWER]: Refusing update from remote peer 1930212695dd4fe8b9c68e351946640d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 10:08:13.320348 12891 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 10:08:13.320710 13312 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 10:08:13.323529 13311 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66ce7cf41bf942779d67d1d64eac375b (127.6.2.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } } }
I20260924 10:08:13.324028 12947 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66ce7cf41bf942779d67d1d64eac375b (127.6.2.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } } }
I20260924 10:08:13.326141 13311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } } }
I20260924 10:08:13.326490 13311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:13.326853 13311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } } }
I20260924 10:08:13.327191 13311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:13.327306 13312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } } }
I20260924 10:08:13.327586 13312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:13.328280 13315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } } }
I20260924 10:08:13.328140 13130 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66ce7cf41bf942779d67d1d64eac375b (127.6.2.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } } }
I20260924 10:08:13.328702 13315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:08:13.331197 13158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1930212695dd4fe8b9c68e351946640d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } } }
I20260924 10:08:13.331430 13158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: This master's current role is: FOLLOWER
W20260924 10:08:13.520593 13310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:13.520852 13310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:13.521018 13310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:13.528175 13310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:13.528278 13310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:13.528463 13310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:08:13.538393 13310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab
--principal=oryx/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:39965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=42237
--tserver_master_addrs=127.6.2.254:41877,127.6.2.253:38781,127.6.2.252:45041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:13.539772 13310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:13.541056 13310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:13.554046 13321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:13.554188 13320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:13.554705 13323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:13.555003 13310 server_base.cc:1034] running on GCE node
I20260924 10:08:13.555600 13310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:13.556509 13310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:13.557760 13310 hybrid_clock.cc:648] HybridClock initialized: now 1790244493557698 us; error 48 us; skew 500 ppm
I20260924 10:08:13.558120 13310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:13 dist-test-slave-s3n3 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244493, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:13.561558 13310 init.cc:377] Logged in from keytab as oryx/127.6.2.194@KRBTEST.COM (short username oryx)
I20260924 10:08:13.563225 13310 webserver.cc:460] Webserver started at http://127.6.2.194:42237/ using document root <none> and password file <none>
I20260924 10:08:13.563972 13310 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:13.564132 13310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:13.569747 13310 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:08:13.572619 13330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:13.574167 13310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 10:08:13.574345 13310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "356ebd95dd6a477ea2216bf6bc20e691"
format_stamp: "Formatted at 2026-09-24 10:08:10 on dist-test-slave-s3n3"
server_key: "a4fc2d3ff7d09c09c3f619a9fd8fd103"
server_key_iv: "93f01024b969c2a8c78f9dd1ea64f9ed"
server_key_version: "encryptionkey@0"
I20260924 10:08:13.575378 13310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:13.591533 13310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:13.595968 13310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:13.596272 13310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:13.597241 13310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:13.598570 13310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:13.598694 13310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:13.598830 13310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:13.598922 13310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:13.633869 13310 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:39965
I20260924 10:08:13.633921 13442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:39965 every 8 connection(s)
I20260924 10:08:13.635383 13310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:08:13.643110 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 13310
I20260924 10:08:13.643462 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 12659
Sep 24 10:08:13 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.2.195@KRBTEST.COM
Sep 24 10:08:13 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244493, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
Sep 24 10:08:13 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244493, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.253@KRBTEST.COM
Sep 24 10:08:13 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244493, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.252@KRBTEST.COM
I20260924 10:08:13.674520 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.644584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.6.2.195:37877)
0924 10:08:13.651696 (+ 7112us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:13.651710 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:13.651753 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:08:13.651849 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:13.651861 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:13.652252 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:13.652465 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:13.655885 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.657517 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.659277 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.660182 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.660320 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.666657 (+ 6337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:13.666673 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:13.666693 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:13.666733 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:13.671032 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.672008 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.672016 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.672019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.672181 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.672641 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.672648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.672654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.673091 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:13.673397 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:13.673740 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 10:08:13.674042 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6885,"thread_start_us":6698,"threads_started":1}
I20260924 10:08:13.676007 12782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:13.681391 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.638891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:45857 (local address 127.6.2.253:38781)
0924 10:08:13.639464 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:13.639471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:13.655672 (+ 16201us) server_negotiation.cc:407] Connection header received
0924 10:08:13.658843 (+ 3171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:13.658849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:13.658941 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:13.659031 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:13.662334 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.663402 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.667024 (+ 3622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.668284 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.668399 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.674142 (+ 5743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:13.674149 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:13.674153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:13.674203 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:13.677546 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.678337 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.678343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.678349 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.678500 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.679229 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.679233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.679236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.679670 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:13.679806 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:13.680791 (+ 985us) server_negotiation.cc:299] Negotiation successful
0924 10:08:13.681018 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":345,"threads_started":1}
I20260924 10:08:13.682646 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.639757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:46441 (local address 127.6.2.254:41877)
0924 10:08:13.650265 (+ 10508us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:13.650275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:13.656864 (+ 6589us) server_negotiation.cc:407] Connection header received
0924 10:08:13.658432 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:13.658438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:13.658531 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:13.658636 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:13.661430 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.662105 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.665960 (+ 3855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.666497 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.666627 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.672028 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:13.672034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:13.672039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:13.672081 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:13.676542 (+ 4461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.677612 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.677619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.677626 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.677855 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.680392 (+ 2537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.680400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.680404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.680747 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:13.680882 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:13.682060 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0924 10:08:13.682283 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10316,"thread_start_us":10201,"threads_started":1}
I20260924 10:08:13.682785 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.639081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:38781 (local address 127.6.2.194:45857)
0924 10:08:13.655547 (+ 16466us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:13.655565 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:13.658111 (+ 2546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:13.659292 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:13.659298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:13.660144 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:13.661234 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:13.661259 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.663833 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.663856 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:13.666763 (+ 2907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:13.666776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.668862 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.668869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:13.669083 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.670197 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:13.670223 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:13.673226 (+ 3003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:13.677693 (+ 4467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:13.677698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:13.677701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:13.678159 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:13.678698 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:13.678703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:13.678708 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:13.679094 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:13.679981 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:13.679989 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:13.680420 (+ 431us) client_negotiation.cc:769] Sending connection context
0924 10:08:13.680704 (+ 284us) client_negotiation.cc:240] Negotiation successful
0924 10:08:13.681232 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16318,"mutex_wait_us":1306,"spinlock_wait_cycles":61952,"thread_start_us":15024,"threads_started":1}
I20260924 10:08:13.683214 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.639786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.194:46441)
0924 10:08:13.656755 (+ 16969us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:13.656782 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:13.658143 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:13.658914 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:13.658926 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:13.660148 (+ 1222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:13.661236 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:13.661261 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.663017 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.663025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:13.665796 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:13.665811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.666742 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.666751 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:13.667055 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.667826 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:13.667865 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:13.671032 (+ 3167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:13.676778 (+ 5746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:13.676787 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:13.676804 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:13.677481 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:13.680115 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:13.680119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:13.680122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:13.680288 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:13.681604 (+ 1316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:13.681609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:13.681744 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 10:08:13.681919 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 10:08:13.682194 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16842,"mutex_wait_us":306,"spinlock_wait_cycles":14720,"thread_start_us":16717,"threads_started":1}
I20260924 10:08:13.683683 13450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.639082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:45041 (local address 127.6.2.194:33331)
0924 10:08:13.654851 (+ 15769us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:13.654915 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:13.658111 (+ 3196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:13.658919 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:13.658928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:13.660144 (+ 1216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:13.661234 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:13.661259 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.662994 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.663000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:13.665791 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:13.665802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.667943 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.667949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:13.668186 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.670887 (+ 2701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:13.670903 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:13.673914 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:13.679475 (+ 5561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:13.679480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:13.679483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:13.680007 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:13.680934 (+ 927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:13.680938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:13.680942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:13.681086 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:13.681744 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:13.681751 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:13.682743 (+ 992us) client_negotiation.cc:769] Sending connection context
0924 10:08:13.682908 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 10:08:13.683230 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15304,"mutex_wait_us":414,"spinlock_wait_cycles":2304,"thread_start_us":16681,"threads_started":1}
I20260924 10:08:13.684247 13447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:13.638559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:33331 (local address 127.6.2.252:45041)
0924 10:08:13.639112 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:13.639119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:13.655087 (+ 15968us) server_negotiation.cc:407] Connection header received
0924 10:08:13.658546 (+ 3459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:13.658552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:13.658640 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:13.658732 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:13.661475 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.662812 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.666982 (+ 4170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:13.667722 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:13.667872 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:13.674977 (+ 7105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:13.674988 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:13.674994 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:13.675050 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:13.679245 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.680128 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.680132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.680134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.680252 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:13.681211 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:13.681216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:13.681220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:13.681556 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:13.681685 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:13.683088 (+ 1403us) server_negotiation.cc:299] Negotiation successful
0924 10:08:13.683323 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":495,"threads_started":1}
I20260924 10:08:13.684401 13443 heartbeater.cc:347] Connected to a master server at 127.6.2.254:41877
I20260924 10:08:13.684527 13445 heartbeater.cc:347] Connected to a master server at 127.6.2.253:38781
I20260924 10:08:13.684762 13443 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:13.684759 13445 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:13.685721 13445 heartbeater.cc:511] Master 127.6.2.253:38781 requested a full tablet report, sending...
I20260924 10:08:13.685750 13443 heartbeater.cc:511] Master 127.6.2.254:41877 requested a full tablet report, sending...
I20260924 10:08:13.686062 13444 heartbeater.cc:347] Connected to a master server at 127.6.2.252:45041
I20260924 10:08:13.686178 13444 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:13.686671 13444 heartbeater.cc:511] Master 127.6.2.252:45041 requested a full tablet report, sending...
I20260924 10:08:13.687683 12937 ts_manager.cc:194] Registered new tserver with Master: 356ebd95dd6a477ea2216bf6bc20e691 (127.6.2.194:39965)
I20260924 10:08:13.687813 12331 ts_manager.cc:194] Re-registered known tserver with Master: 356ebd95dd6a477ea2216bf6bc20e691 (127.6.2.194:39965)
I20260924 10:08:13.688869 12331 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:46441
I20260924 10:08:13.689909 13120 ts_manager.cc:194] Registered new tserver with Master: 356ebd95dd6a477ea2216bf6bc20e691 (127.6.2.194:39965)
I20260924 10:08:13.765573 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 12659
I20260924 10:08:13.782514 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:37877
--local_ip_for_outbound_sockets=127.6.2.195
--tserver_master_addrs=127.6.2.254:41877
--webserver_port=46637
--webserver_interface=127.6.2.195
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab
--principal=oryx/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.2.254:41877,127.6.2.253:38781
--tserver_master_addrs=127.6.2.254:41877,127.6.2.253:38781,127.6.2.252:45041 with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:08:14.080724 13298 heartbeater.cc:503] Master 127.6.2.254:41877 was elected leader, sending a full tablet report...
W20260924 10:08:14.116333 13456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:14.116613 13456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:14.116710 13456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:14.124053 13456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:14.124167 13456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:14.124280 13456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:08:14.133982 13456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab
--principal=oryx/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:37877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=46637
--tserver_master_addrs=127.6.2.254:41877,127.6.2.253:38781,127.6.2.252:45041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:14.135272 13456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:14.136605 13456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:14.149616 13464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:14.149607 13461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:14.149858 13462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:14.155184 13456 server_base.cc:1034] running on GCE node
I20260924 10:08:14.155859 13456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:14.156829 13456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:14.158048 13456 hybrid_clock.cc:648] HybridClock initialized: now 1790244494157980 us; error 56 us; skew 500 ppm
I20260924 10:08:14.158341 13456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:14 dist-test-slave-s3n3 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244494, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:14.161649 13456 init.cc:377] Logged in from keytab as oryx/127.6.2.195@KRBTEST.COM (short username oryx)
I20260924 10:08:14.163475 13456 webserver.cc:460] Webserver started at http://127.6.2.195:46637/ using document root <none> and password file <none>
I20260924 10:08:14.164251 13456 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:14.164408 13456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:14.174561 13456 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.005s sys 0.004s
I20260924 10:08:14.177670 13471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:14.179236 13456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:08:14.179394 13456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "7258ad39bc4a4b29ab53cd2483c8c591"
format_stamp: "Formatted at 2026-09-24 10:08:10 on dist-test-slave-s3n3"
server_key: "6fcd123eaacb67e33afdfac00af72f8e"
server_key_iv: "5a830690da46b959fb9cecafa679404e"
server_key_version: "encryptionkey@0"
I20260924 10:08:14.180420 13456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:14.200328 13456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:14.204717 13456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:14.204952 13456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:14.205858 13456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:14.207187 13456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:14.207273 13456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:14.207352 13456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:14.207412 13456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:14.240960 13456 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:37877
I20260924 10:08:14.241045 13583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:37877 every 8 connection(s)
I20260924 10:08:14.242372 13456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:08:14.244071 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 13456
Sep 24 10:08:14 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244494, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.252@KRBTEST.COM
Sep 24 10:08:14 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244494, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
Sep 24 10:08:14 dist-test-slave-s3n3 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244494, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.253@KRBTEST.COM
I20260924 10:08:14.315089 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.271009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.6.2.253:38781)
0924 10:08:14.294088 (+ 23079us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.294094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.295819 (+ 1725us) server_negotiation.cc:407] Connection header received
0924 10:08:14.295878 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.295883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.295965 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.296067 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:14.298296 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.299557 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.301463 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.302135 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.302270 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.306006 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:14.306012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:14.306017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:14.306056 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:14.310298 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.312025 (+ 1727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.312031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.312035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.312191 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.312682 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.312694 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.312698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.313095 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:14.313386 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.314430 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.314681 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22948,"thread_start_us":39869,"threads_started":1}
I20260924 10:08:14.320593 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.260909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:45393 (local address 127.6.2.253:38781)
0924 10:08:14.262715 (+ 1806us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.262725 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.267627 (+ 4902us) server_negotiation.cc:407] Connection header received
0924 10:08:14.267975 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.267981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.268056 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.268180 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:14.292670 (+ 24490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.294214 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.307191 (+ 12977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.307959 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.308068 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.312600 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:14.312608 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:14.312615 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:14.312656 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:14.316538 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.317618 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.317626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.317630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.317787 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.318310 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.318318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.318324 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.318689 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:14.318835 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.319945 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.320154 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1658,"thread_start_us":1522,"threads_started":1}
I20260924 10:08:14.321916 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.260356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:38781 (local address 127.6.2.195:45393)
0924 10:08:14.266919 (+ 6563us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:14.266940 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:14.267529 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:14.268528 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:14.268545 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:14.270362 (+ 1817us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:14.292467 (+ 22105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.292480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.297492 (+ 5012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.297500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.299678 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.299696 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.308440 (+ 8744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.308448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.308660 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.309187 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:14.309210 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:14.312347 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:14.316706 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:14.316717 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:14.316737 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:14.317471 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:14.317944 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:14.317948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:14.317952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:14.318173 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:14.318937 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:14.318946 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:14.319492 (+ 546us) client_negotiation.cc:769] Sending connection context
0924 10:08:14.319831 (+ 339us) client_negotiation.cc:240] Negotiation successful
0924 10:08:14.320431 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6449,"thread_start_us":710,"threads_started":1}
I20260924 10:08:14.322275 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.245396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:53609 (local address 127.6.2.254:41877)
0924 10:08:14.267646 (+ 22250us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.267655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.267684 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:08:14.268806 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.268814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.268925 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.269060 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:14.292606 (+ 23546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.294167 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.297311 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.298119 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.298254 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.310185 (+ 11931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:14.310192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:14.310197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:14.310239 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:14.314276 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.320220 (+ 5944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.320226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.320229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.320359 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.320870 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.320875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.320878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.321226 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:14.321322 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.321745 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.321947 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22079,"thread_start_us":21981,"threads_started":1}
I20260924 10:08:14.322366 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.252545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.195:53609)
0924 10:08:14.261288 (+ 8743us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:14.261349 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:14.267558 (+ 6209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:14.269537 (+ 1979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:14.269544 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:14.285820 (+ 16276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:14.292124 (+ 6304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.292140 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.294453 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.294462 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.297126 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.297156 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.302441 (+ 5285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.302447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.302665 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.305233 (+ 2568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:14.305260 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:14.309920 (+ 4660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:14.318900 (+ 8980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:14.318905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:14.318909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:14.319450 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:14.320505 (+ 1055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:14.320509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:14.320513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:14.320708 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:14.321382 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:14.321386 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:14.321506 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 10:08:14.321636 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 10:08:14.321928 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8345,"thread_start_us":13894,"threads_started":1}
I20260924 10:08:14.323765 13586 heartbeater.cc:347] Connected to a master server at 127.6.2.253:38781
I20260924 10:08:14.323947 13584 heartbeater.cc:347] Connected to a master server at 127.6.2.254:41877
I20260924 10:08:14.324103 13586 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:14.324106 13584 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:14.325088 13584 heartbeater.cc:511] Master 127.6.2.254:41877 requested a full tablet report, sending...
I20260924 10:08:14.325426 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.245439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:56047 (local address 127.6.2.252:45041)
0924 10:08:14.256006 (+ 10567us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.256016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.266042 (+ 10026us) server_negotiation.cc:407] Connection header received
0924 10:08:14.267960 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.267969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.268082 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.268173 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:14.286793 (+ 18620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.288444 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.299902 (+ 11458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.300826 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.300950 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.311008 (+ 10058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:14.311015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:14.311021 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:14.311101 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:14.317240 (+ 6139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.322648 (+ 5408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.322654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.322659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.323157 (+ 498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.323622 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.323627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.323631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.324119 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:14.324291 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.324833 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.325046 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10373,"thread_start_us":10244,"threads_started":1}
I20260924 10:08:14.326116 13586 heartbeater.cc:511] Master 127.6.2.253:38781 requested a full tablet report, sending...
I20260924 10:08:14.326262 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.276191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.6.2.252:45041)
0924 10:08:14.290446 (+ 14255us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.290454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.299020 (+ 8566us) server_negotiation.cc:407] Connection header received
0924 10:08:14.299115 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.299122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.299196 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.299290 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:14.306890 (+ 7600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.308590 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.314161 (+ 5571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.314911 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.315039 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.318421 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:14.318427 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:14.318431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:14.318466 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:14.321744 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.323184 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.323191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.323194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.323361 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.323795 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.323801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.323806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.324137 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:14.324378 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.325673 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.325873 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14073,"thread_start_us":13952,"threads_started":1}
I20260924 10:08:14.326643 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.253227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:45041 (local address 127.6.2.195:56047)
0924 10:08:14.265895 (+ 12668us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:14.265920 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:14.267728 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:14.268526 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:14.268545 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:14.285118 (+ 16573us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:14.286452 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.286498 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.288726 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.288736 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.291775 (+ 3039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.291799 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.301392 (+ 9593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.301402 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.301844 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.303194 (+ 1350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:14.303232 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:14.310782 (+ 7550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:14.319570 (+ 8788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:14.319574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:14.319578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:14.320068 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:14.323306 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:14.323313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:14.323320 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:14.323491 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:14.324381 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:14.324386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:14.324535 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 10:08:14.325666 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0924 10:08:14.326230 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12493,"thread_start_us":12078,"threads_started":1}
I20260924 10:08:14.327008 12331 ts_manager.cc:194] Re-registered known tserver with Master: 7258ad39bc4a4b29ab53cd2483c8c591 (127.6.2.195:37877)
I20260924 10:08:14.327557 13585 heartbeater.cc:347] Connected to a master server at 127.6.2.252:45041
I20260924 10:08:14.327662 13585 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:14.328088 13585 heartbeater.cc:511] Master 127.6.2.252:45041 requested a full tablet report, sending...
I20260924 10:08:14.328171 12331 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:53609
I20260924 10:08:14.328797 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.271763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.6.2.254:41877)
0924 10:08:14.294742 (+ 22979us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.294750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.294777 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:08:14.294838 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.294845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.294941 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.295198 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:14.300272 (+ 5074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.301403 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.305070 (+ 3667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.305756 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.305886 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.316194 (+ 10308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:14.316200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:14.316205 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:14.316245 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:14.319597 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.323342 (+ 3745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.323347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.323351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.323491 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.324892 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.324896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.324900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.325269 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:14.325874 (+ 605us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.328193 (+ 2319us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.328419 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22829,"thread_start_us":23580,"threads_started":1}
I20260924 10:08:14.329316 12937 ts_manager.cc:194] Registered new tserver with Master: 7258ad39bc4a4b29ab53cd2483c8c591 (127.6.2.195:37877)
I20260924 10:08:14.329623 13119 ts_manager.cc:194] Registered new tserver with Master: 7258ad39bc4a4b29ab53cd2483c8c591 (127.6.2.195:37877)
I20260924 10:08:14.332741 12331 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:08:14.339624 12331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:08:14.368865 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.356619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:39965 (local address 127.0.0.1:44688)
0924 10:08:14.357167 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:14.357187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:14.357310 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:14.357844 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:14.357849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:14.357873 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:14.358807 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.358824 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.361324 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.361332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.364423 (+ 3091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.364444 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.367563 (+ 3119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.367572 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.367936 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.368030 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 10:08:14.368149 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 10:08:14.368358 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":250,"threads_started":1}
I20260924 10:08:14.370380 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.356800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.6.2.194:39965)
0924 10:08:14.357292 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.357301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.357330 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:08:14.357460 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.357472 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.357672 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.357868 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:14.359091 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.361093 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.364675 (+ 3582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.366934 (+ 2259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.367100 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.367382 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.369748 (+ 2366us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.370000 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":218,"threads_started":1}
I20260924 10:08:14.374130 13378 tablet_service.cc:1467] Processing CreateTablet for tablet b47ea727ee1f40fda5c3be2747e77025 (DEFAULT_TABLE table=test-table [id=cd439325fabd428d87f894255e9012f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:14.375445 13378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47ea727ee1f40fda5c3be2747e77025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:14.378793 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.357866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:45265 (local address 127.0.0.1:35666)
0924 10:08:14.365315 (+ 7449us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:14.365346 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:14.365489 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:14.371918 (+ 6429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:14.371927 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:14.371977 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:14.372408 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.372422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.374895 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.374904 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.376677 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.376686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.377954 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.377960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.378137 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.378171 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:08:14.378277 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 10:08:14.378416 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":377,"threads_started":1}
I20260924 10:08:14.380673 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.358306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.6.2.193:45265)
0924 10:08:14.371089 (+ 12783us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.371102 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.371141 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:08:14.371315 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.371326 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.371654 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.372634 (+ 980us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:14.372890 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.374706 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.376825 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.377739 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.377891 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.378114 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.380331 (+ 2217us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.380460 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12524,"thread_start_us":12400,"threads_started":1}
I20260924 10:08:14.382841 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.358350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:37877 (local address 127.0.0.1:45918)
0924 10:08:14.369375 (+ 11025us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:14.369399 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:14.369517 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:14.376221 (+ 6704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:14.376227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:14.376260 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:14.376415 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.376425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.378248 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.378254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.380114 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.380131 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.382047 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.382053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.382232 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.382274 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 10:08:14.382381 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 10:08:14.382525 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10888,"mutex_wait_us":39,"thread_start_us":12039,"threads_started":1}
I20260924 10:08:14.383113 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.358766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.6.2.195:37877)
0924 10:08:14.375643 (+ 16877us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.375661 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.375692 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:08:14.375784 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.375791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.376022 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.376209 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:14.376578 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.378110 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.380308 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.381864 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.381999 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.382199 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.382418 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.382573 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16660,"thread_start_us":16497,"threads_started":1}
I20260924 10:08:14.384320 13233 tablet_service.cc:1467] Processing CreateTablet for tablet b47ea727ee1f40fda5c3be2747e77025 (DEFAULT_TABLE table=test-table [id=cd439325fabd428d87f894255e9012f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:14.385772 13233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47ea727ee1f40fda5c3be2747e77025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:14.386693 13519 tablet_service.cc:1467] Processing CreateTablet for tablet b47ea727ee1f40fda5c3be2747e77025 (DEFAULT_TABLE table=test-table [id=cd439325fabd428d87f894255e9012f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:14.388300 13519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47ea727ee1f40fda5c3be2747e77025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:14.390206 13621 tablet_bootstrap.cc:492] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691: Bootstrap starting.
I20260924 10:08:14.393100 13621 tablet_bootstrap.cc:654] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:14.394076 13621 log.cc:824] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:14.396982 13621 tablet_bootstrap.cc:492] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691: No bootstrap required, opened a new log
I20260924 10:08:14.397246 13621 ts_tablet_manager.cc:1397] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 10:08:14.401176 13623 tablet_bootstrap.cc:492] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f: Bootstrap starting.
I20260924 10:08:14.403046 13624 tablet_bootstrap.cc:492] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591: Bootstrap starting.
I20260924 10:08:14.404805 13623 tablet_bootstrap.cc:654] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:14.405728 13624 tablet_bootstrap.cc:654] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:14.406033 13623 log.cc:824] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:14.406203 13621 raft_consensus.cc:348] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } } peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } }
I20260924 10:08:14.406702 13621 raft_consensus.cc:374] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:14.406836 13621 raft_consensus.cc:729] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356ebd95dd6a477ea2216bf6bc20e691, State: Initialized, Role: FOLLOWER
I20260924 10:08:14.407627 13624 log.cc:824] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:14.407635 13621 consensus_queue.cc:260] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } } peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } }
I20260924 10:08:14.410318 13624 tablet_bootstrap.cc:492] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591: No bootstrap required, opened a new log
I20260924 10:08:14.410584 13624 ts_tablet_manager.cc:1397] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 10:08:14.411890 13443 heartbeater.cc:503] Master 127.6.2.254:41877 was elected leader, sending a full tablet report...
I20260924 10:08:14.413091 13621 ts_tablet_manager.cc:1428] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260924 10:08:14.420394 13624 raft_consensus.cc:348] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } } peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } }
I20260924 10:08:14.420861 13624 raft_consensus.cc:374] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:14.420976 13624 raft_consensus.cc:729] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7258ad39bc4a4b29ab53cd2483c8c591, State: Initialized, Role: FOLLOWER
I20260924 10:08:14.421671 13623 tablet_bootstrap.cc:492] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f: No bootstrap required, opened a new log
I20260924 10:08:14.421590 13624 consensus_queue.cc:260] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } } peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } }
I20260924 10:08:14.421980 13623 ts_tablet_manager.cc:1397] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.009s
I20260924 10:08:14.422782 13584 heartbeater.cc:503] Master 127.6.2.254:41877 was elected leader, sending a full tablet report...
I20260924 10:08:14.423377 13624 ts_tablet_manager.cc:1428] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 10:08:14.430282 13623 raft_consensus.cc:348] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } } peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } }
I20260924 10:08:14.430650 13623 raft_consensus.cc:374] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:14.430758 13623 raft_consensus.cc:729] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe69c95bbae4afaaf1af62cbb2e918f, State: Initialized, Role: FOLLOWER
I20260924 10:08:14.431351 13623 consensus_queue.cc:260] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } } peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } }
I20260924 10:08:14.432928 13623 ts_tablet_manager.cc:1428] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260924 10:08:14.532289 13301 tablet.cc:2369] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:08:14.557451 13592 tablet.cc:2369] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:08:14.638978 13446 tablet.cc:2369] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:08:14.648749 13627 raft_consensus.cc:482] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:08:14.649058 13627 raft_consensus.cc:504] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } } peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } }
I20260924 10:08:14.650981 13627 leader_election.cc:288] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbe69c95bbae4afaaf1af62cbb2e918f (127.6.2.193:45265), 7258ad39bc4a4b29ab53cd2483c8c591 (127.6.2.195:37877)
I20260924 10:08:14.655838 13629 raft_consensus.cc:482] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:08:14.656122 13629 raft_consensus.cc:504] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } } peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } }
I20260924 10:08:14.657816 13629 leader_election.cc:288] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 356ebd95dd6a477ea2216bf6bc20e691 (127.6.2.194:39965), 7258ad39bc4a4b29ab53cd2483c8c591 (127.6.2.195:37877)
I20260924 10:08:14.658790 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.651462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:46801 (local address 127.6.2.195:37877)
0924 10:08:14.651725 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.651732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.652342 (+ 610us) server_negotiation.cc:407] Connection header received
0924 10:08:14.652537 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.652544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.652623 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.652690 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:14.653391 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.654158 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.656307 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.657570 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.657698 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.657803 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.658388 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.658538 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:08:14.658797 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.651297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:59407 (local address 127.6.2.193:45265)
0924 10:08:14.651510 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.651517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.652429 (+ 912us) server_negotiation.cc:407] Connection header received
0924 10:08:14.652603 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.652607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.652794 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.652883 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:14.653487 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.654210 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.656242 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.657578 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.657698 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.657818 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.658410 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.658538 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:08:14.659631 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.651285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:37877 (local address 127.6.2.194:46801)
0924 10:08:14.652236 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:14.652255 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:14.652368 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:14.652826 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:14.652834 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:14.652868 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:14.653227 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.653239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.654295 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.654302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.656092 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.656106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.657985 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.657994 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.658177 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.658213 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:08:14.659023 (+ 810us) client_negotiation.cc:240] Negotiation successful
0924 10:08:14.659197 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":40,"thread_start_us":451,"threads_started":1}
I20260924 10:08:14.660180 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.651352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:45265 (local address 127.6.2.194:59407)
0924 10:08:14.652291 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:14.652326 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:14.652463 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:14.652932 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:14.652936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:14.652958 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:14.653271 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.653278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.654363 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.654367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.656086 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.656101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.658086 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.658091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.658235 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.658275 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 10:08:14.658934 (+ 659us) client_negotiation.cc:240] Negotiation successful
0924 10:08:14.659821 (+ 887us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":744,"thread_start_us":457,"threads_started":1}
I20260924 10:08:14.661245 13253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47ea727ee1f40fda5c3be2747e77025" candidate_uuid: "356ebd95dd6a477ea2216bf6bc20e691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" is_pre_election: true
I20260924 10:08:14.661762 13253 raft_consensus.cc:2454] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 356ebd95dd6a477ea2216bf6bc20e691 in term 0.
I20260924 10:08:14.662793 13332 leader_election.cc:302] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 356ebd95dd6a477ea2216bf6bc20e691, cbe69c95bbae4afaaf1af62cbb2e918f; no voters:
I20260924 10:08:14.663424 13627 raft_consensus.cc:2781] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:08:14.663592 13627 raft_consensus.cc:482] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:08:14.663694 13627 raft_consensus.cc:3037] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:14.668159 13627 raft_consensus.cc:504] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } } peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } }
I20260924 10:08:14.669524 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.659197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:48147 (local address 127.6.2.194:39965)
0924 10:08:14.659370 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.659376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.660091 (+ 715us) server_negotiation.cc:407] Connection header received
0924 10:08:14.660497 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.660502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.660569 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.660632 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:14.661216 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.662559 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.665985 (+ 3426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.667830 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.667955 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.668127 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.668992 (+ 865us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.669185 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:08:14.669664 13253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47ea727ee1f40fda5c3be2747e77025" candidate_uuid: "356ebd95dd6a477ea2216bf6bc20e691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f"
I20260924 10:08:14.669878 13253 raft_consensus.cc:3037] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:14.670053 13539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47ea727ee1f40fda5c3be2747e77025" candidate_uuid: "356ebd95dd6a477ea2216bf6bc20e691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" is_pre_election: true
I20260924 10:08:14.670382 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.658118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:39965 (local address 127.6.2.193:48147)
0924 10:08:14.660000 (+ 1882us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:14.660025 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:14.660194 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:14.660796 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:14.660799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:14.660818 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:14.661087 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.661094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.662761 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.662769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.665761 (+ 2992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.665781 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.668426 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.668435 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.668762 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.668817 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 10:08:14.669702 (+ 885us) client_negotiation.cc:240] Negotiation successful
0924 10:08:14.669912 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1717,"threads_started":1}
I20260924 10:08:14.670610 13539 raft_consensus.cc:2454] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 356ebd95dd6a477ea2216bf6bc20e691 in term 0.
I20260924 10:08:14.672062 13627 leader_election.cc:288] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [CANDIDATE]: Term 1 election: Requested vote from peers cbe69c95bbae4afaaf1af62cbb2e918f (127.6.2.193:45265), 7258ad39bc4a4b29ab53cd2483c8c591 (127.6.2.195:37877)
I20260924 10:08:14.672384 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47ea727ee1f40fda5c3be2747e77025" candidate_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "356ebd95dd6a477ea2216bf6bc20e691" is_pre_election: true
I20260924 10:08:14.672570 13539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47ea727ee1f40fda5c3be2747e77025" candidate_uuid: "356ebd95dd6a477ea2216bf6bc20e691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7258ad39bc4a4b29ab53cd2483c8c591"
I20260924 10:08:14.672770 13539 raft_consensus.cc:3037] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:14.672864 13398 raft_consensus.cc:2379] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cbe69c95bbae4afaaf1af62cbb2e918f in current term 1: Already voted for candidate 356ebd95dd6a477ea2216bf6bc20e691 in this term.
I20260924 10:08:14.673581 13253 raft_consensus.cc:2454] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 356ebd95dd6a477ea2216bf6bc20e691 in term 1.
I20260924 10:08:14.674015 13332 leader_election.cc:302] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 356ebd95dd6a477ea2216bf6bc20e691, cbe69c95bbae4afaaf1af62cbb2e918f; no voters:
I20260924 10:08:14.674345 13627 raft_consensus.cc:2781] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:14.674784 13627 raft_consensus.cc:686] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 1 LEADER]: Becoming Leader. State: Replica: 356ebd95dd6a477ea2216bf6bc20e691, State: Running, Role: LEADER
I20260924 10:08:14.675602 13627 consensus_queue.cc:237] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } } peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } }
I20260924 10:08:14.675805 13539 raft_consensus.cc:2454] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 356ebd95dd6a477ea2216bf6bc20e691 in term 1.
I20260924 10:08:14.677471 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.658216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:37877 (local address 127.6.2.193:57647)
0924 10:08:14.670657 (+ 12441us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:14.670682 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:14.670922 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:14.671503 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:14.671507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:14.671536 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:14.671741 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.671754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.673327 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.673332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.675227 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:14.675240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.676771 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.676776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:14.676989 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.677040 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 10:08:14.677136 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 10:08:14.677260 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12284,"thread_start_us":1314,"threads_started":1}
I20260924 10:08:14.677572 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.658569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:57647 (local address 127.6.2.195:37877)
0924 10:08:14.658991 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.658996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.670766 (+ 11770us) server_negotiation.cc:407] Connection header received
0924 10:08:14.671121 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.671129 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.671225 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.671292 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:14.671923 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.673124 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.675422 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.676617 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.676741 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.676838 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.677181 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.677308 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":10542,"threads_started":1}
I20260924 10:08:14.678172 13539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47ea727ee1f40fda5c3be2747e77025" candidate_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" is_pre_election: true
I20260924 10:08:14.678427 13539 raft_consensus.cc:2379] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cbe69c95bbae4afaaf1af62cbb2e918f in current term 1: Already voted for candidate 356ebd95dd6a477ea2216bf6bc20e691 in this term.
I20260924 10:08:14.679106 13188 leader_election.cc:302] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cbe69c95bbae4afaaf1af62cbb2e918f; no voters: 356ebd95dd6a477ea2216bf6bc20e691, 7258ad39bc4a4b29ab53cd2483c8c591
I20260924 10:08:14.679605 13629 raft_consensus.cc:2726] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 10:08:14.681759 12331 catalog_manager.cc:5495] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 reported cstate change: term changed from 0 to 1, leader changed from <none> to 356ebd95dd6a477ea2216bf6bc20e691 (127.6.2.194). New cstate: current_term: 1 leader_uuid: "356ebd95dd6a477ea2216bf6bc20e691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } health_report { overall_health: UNKNOWN } } }
I20260924 10:08:14.733383 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.728167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.6.2.194:39965)
0924 10:08:14.728332 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.728336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.728460 (+ 124us) server_negotiation.cc:407] Connection header received
0924 10:08:14.728633 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.728637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.728738 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.728871 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:08:14.729173 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.729975 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.731304 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.731729 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.731831 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.732292 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:08:14.732847 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:08:14.732966 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.733080 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.733147 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:08:14.741390 13539 raft_consensus.cc:1261] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 1 FOLLOWER]: Refusing update from remote peer 356ebd95dd6a477ea2216bf6bc20e691: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:14.741434 13253 raft_consensus.cc:1261] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [term 1 FOLLOWER]: Refusing update from remote peer 356ebd95dd6a477ea2216bf6bc20e691: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:14.742417 13627 consensus_queue.cc:1035] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:14.742826 13635 consensus_queue.cc:1035] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:14.757721 13638 mvcc.cc:205] Tried to move back new op lower bound from 7332841450450518016 to 7332841450195636224. Current Snapshot: MvccSnapshot[applied={T|T < 7332841450450518016}]
I20260924 10:08:14.776296 12330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:14.776654 12330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:14.782719 12330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1930212695dd4fe8b9c68e351946640d: Sending DeleteTablet for 3 replicas of tablet b47ea727ee1f40fda5c3be2747e77025
I20260924 10:08:14.784090 13233 tablet_service.cc:1514] Processing DeleteTablet for tablet b47ea727ee1f40fda5c3be2747e77025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:14 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:35666
I20260924 10:08:14.785386 13378 tablet_service.cc:1514] Processing DeleteTablet for tablet b47ea727ee1f40fda5c3be2747e77025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:14 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:44688
I20260924 10:08:14.785424 13519 tablet_service.cc:1514] Processing DeleteTablet for tablet b47ea727ee1f40fda5c3be2747e77025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:14 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:45918
I20260924 10:08:14.794250 13649 tablet_replica.cc:330] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691: stopping tablet replica
I20260924 10:08:14.794845 13649 raft_consensus.cc:2229] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:08:14.795557 13649 raft_consensus.cc:2258] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:14.797088 13647 tablet_replica.cc:330] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f: stopping tablet replica
I20260924 10:08:14.797508 13647 raft_consensus.cc:2229] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:14.798000 13647 raft_consensus.cc:2258] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:14.798632 13649 ts_tablet_manager.cc:1905] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:14.799933 13648 tablet_replica.cc:330] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591: stopping tablet replica
I20260924 10:08:14.800161 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 12299
I20260924 10:08:14.800331 13648 raft_consensus.cc:2229] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:14.800807 13648 raft_consensus.cc:2258] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:14.801183 12361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:14.803205 13648 ts_tablet_manager.cc:1905] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:14.804322 13647 ts_tablet_manager.cc:1905] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:14.811208 13648 ts_tablet_manager.cc:1918] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:14.811460 13648 log.cc:1196] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/b47ea727ee1f40fda5c3be2747e77025
I20260924 10:08:14.812252 13648 ts_tablet_manager.cc:1939] T b47ea727ee1f40fda5c3be2747e77025 P 7258ad39bc4a4b29ab53cd2483c8c591: Deleting consensus metadata
I20260924 10:08:14.818605 13647 ts_tablet_manager.cc:1918] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:14.818776 13647 log.cc:1196] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/b47ea727ee1f40fda5c3be2747e77025
I20260924 10:08:14.819398 13647 ts_tablet_manager.cc:1939] T b47ea727ee1f40fda5c3be2747e77025 P cbe69c95bbae4afaaf1af62cbb2e918f: Deleting consensus metadata
I20260924 10:08:14.820649 13649 ts_tablet_manager.cc:1918] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:14.820825 13649 log.cc:1196] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/b47ea727ee1f40fda5c3be2747e77025
I20260924 10:08:14.821393 13649 ts_tablet_manager.cc:1939] T b47ea727ee1f40fda5c3be2747e77025 P 356ebd95dd6a477ea2216bf6bc20e691: Deleting consensus metadata
I20260924 10:08:14.824728 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.789092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.6.2.253:38781)
0924 10:08:14.789264 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.789271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.798862 (+ 9591us) server_negotiation.cc:407] Connection header received
0924 10:08:14.799323 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.799331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.799433 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.799543 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:14.807618 (+ 8075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.808783 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.810705 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.811439 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.811586 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.815144 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:14.815150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:14.815156 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:14.815199 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:14.819148 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.821863 (+ 2715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.821868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.821871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.822034 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.822593 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.822600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.822605 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.823037 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:14.823374 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.823890 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.824231 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:08:14.843161 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.799016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.6.2.252:45041)
0924 10:08:14.799234 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.799240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.824678 (+ 25438us) server_negotiation.cc:407] Connection header received
0924 10:08:14.824997 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.825003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.825106 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.825180 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:14.827262 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.828348 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.830273 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.830934 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:14.831037 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:14.834780 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:14.834787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:14.834792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:14.834834 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:14.838979 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.840515 (+ 1536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.840522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.840526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.840710 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:14.841238 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:14.841244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:14.841247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:14.841613 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:14.841880 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:14.842441 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 10:08:14.842705 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 10:08:14.952345 12316 catalog_manager.cc:4841] TS cbe69c95bbae4afaaf1af62cbb2e918f (127.6.2.193:45265): tablet b47ea727ee1f40fda5c3be2747e77025 (table test-table [id=cd439325fabd428d87f894255e9012f3]) successfully deleted
W20260924 10:08:14.952713 12316 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.2.193:45265, user_credentials={real_user=slave}} blocked reactor thread for 131405us
I20260924 10:08:14.953248 12316 catalog_manager.cc:4841] TS 356ebd95dd6a477ea2216bf6bc20e691 (127.6.2.194:39965): tablet b47ea727ee1f40fda5c3be2747e77025 (table test-table [id=cd439325fabd428d87f894255e9012f3]) successfully deleted
I20260924 10:08:14.953589 12317 catalog_manager.cc:4841] TS 7258ad39bc4a4b29ab53cd2483c8c591 (127.6.2.195:37877): tablet b47ea727ee1f40fda5c3be2747e77025 (table test-table [id=cd439325fabd428d87f894255e9012f3]) successfully deleted
W20260924 10:08:14.953788 12317 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.2.195:37877, user_credentials={real_user=slave}} blocked reactor thread for 139399us
I20260924 10:08:14.954207 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 12299
I20260924 10:08:15.013557 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.982710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.6.2.252:45041)
0924 10:08:14.983242 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.983252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:14.993021 (+ 9769us) server_negotiation.cc:407] Connection header received
0924 10:08:14.993648 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:14.993657 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:14.993797 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:14.993896 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:14.996896 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:14.998407 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:15.002198 (+ 3791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:15.002739 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:15.002879 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:15.006350 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:15.006355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:15.006359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:15.006392 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:15.009917 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:15.010795 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:15.010802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:15.010807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:15.010975 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:15.011393 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:15.011398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:15.011403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:15.011789 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:15.012046 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:15.012923 (+ 877us) server_negotiation.cc:299] Negotiation successful
0924 10:08:15.013175 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290}
I20260924 10:08:15.016987 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:14.982927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.6.2.253:38781)
0924 10:08:14.983418 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:14.983424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:15.001708 (+ 18284us) server_negotiation.cc:407] Connection header received
0924 10:08:15.002185 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:15.002190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:15.002277 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:15.002377 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:15.004626 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:15.005823 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:15.007639 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:15.008307 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:15.008432 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:15.011106 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:15.011114 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:15.011119 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:15.011162 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:15.014349 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:15.015139 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:15.015145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:15.015147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:15.015259 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:15.015615 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:15.015620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:15.015623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:15.015909 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:15.016144 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:15.016461 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 10:08:15.016674 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350}
I20260924 10:08:15.764612 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:15.763765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.193:53615)
0924 10:08:15.764245 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:15.764371 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":257,"threads_started":1}
W20260924 10:08:15.765064 13298 heartbeater.cc:644] Failed to heartbeat to 127.6.2.254:41877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
I20260924 10:08:15.769531 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:15.768682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.194:52075)
0924 10:08:15.769204 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:15.769335 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":282,"threads_started":1}
W20260924 10:08:15.769995 13443 heartbeater.cc:644] Failed to heartbeat to 127.6.2.254:41877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
I20260924 10:08:15.771548 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:15.770743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.195:56121)
0924 10:08:15.771209 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:15.771349 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":224,"threads_started":1}
W20260924 10:08:15.772040 13584 heartbeater.cc:644] Failed to heartbeat to 127.6.2.254:41877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
I20260924 10:08:16.283910 13662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1930212695dd4fe8b9c68e351946640d)
I20260924 10:08:16.284163 13662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } }
I20260924 10:08:16.285040 13662 leader_election.cc:288] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1930212695dd4fe8b9c68e351946640d (127.6.2.254:41877), 9dd49fc19ab84da6aa61722da0b2afc0 (127.6.2.253:38781)
I20260924 10:08:16.286554 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.285287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:60492)
0924 10:08:16.285888 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.286200 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":718,"threads_started":1}
W20260924 10:08:16.286826 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
I20260924 10:08:16.286952 13666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1930212695dd4fe8b9c68e351946640d)
I20260924 10:08:16.287240 13666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } }
I20260924 10:08:16.288820 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.288479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:60508)
0924 10:08:16.288607 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.288712 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 10:08:16.289027 13103 leader_election.cc:334] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1930212695dd4fe8b9c68e351946640d (127.6.2.254:41877): Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
I20260924 10:08:16.289806 13666 leader_election.cc:288] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1930212695dd4fe8b9c68e351946640d (127.6.2.254:41877), 66ce7cf41bf942779d67d1d64eac375b (127.6.2.252:45041)
I20260924 10:08:16.290741 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.289265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:60510)
0924 10:08:16.290317 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.290504 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":933,"thread_start_us":252,"threads_started":1}
W20260924 10:08:16.290966 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
I20260924 10:08:16.295993 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.285307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:38781 (local address 127.0.0.1:56764)
0924 10:08:16.286218 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.286245 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:16.286410 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:16.287537 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:16.287542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:16.287637 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:16.287996 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:16.288014 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.289418 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.289425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:16.292110 (+ 2685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:16.292130 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.294785 (+ 2655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.294793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:16.295049 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.295136 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 10:08:16.295292 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 10:08:16.295504 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":263,"threads_started":1}
I20260924 10:08:16.296007 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.285433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56764 (local address 127.6.2.253:38781)
0924 10:08:16.287165 (+ 1732us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:16.287171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:16.287196 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:08:16.287273 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:16.287276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:16.287332 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:16.287451 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:16.288179 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.289286 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.292328 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.294150 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.294288 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.294422 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:16.295332 (+ 910us) server_negotiation.cc:299] Negotiation successful
0924 10:08:16.295543 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1509,"thread_start_us":1244,"threads_started":1}
I20260924 10:08:16.299903 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66ce7cf41bf942779d67d1d64eac375b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" is_pre_election: true
I20260924 10:08:16.300262 12947 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ce7cf41bf942779d67d1d64eac375b in term 1.
I20260924 10:08:16.300894 13106 leader_election.cc:302] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66ce7cf41bf942779d67d1d64eac375b, 9dd49fc19ab84da6aa61722da0b2afc0; no voters: 1930212695dd4fe8b9c68e351946640d
I20260924 10:08:16.301231 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.300858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:60536)
0924 10:08:16.301003 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.301086 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 10:08:16.301258 13662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 10:08:16.301394 13662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1930212695dd4fe8b9c68e351946640d)
I20260924 10:08:16.301483 13662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 1 FOLLOWER]: Advancing to term 2
W20260924 10:08:16.301573 12921 leader_election.cc:334] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1930212695dd4fe8b9c68e351946640d (127.6.2.254:41877): Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
I20260924 10:08:16.303961 13662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } }
I20260924 10:08:16.304697 13662 leader_election.cc:288] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [CANDIDATE]: Term 2 election: Requested vote from peers 1930212695dd4fe8b9c68e351946640d (127.6.2.254:41877), 9dd49fc19ab84da6aa61722da0b2afc0 (127.6.2.253:38781)
I20260924 10:08:16.305254 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.304829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:60546)
0924 10:08:16.304995 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.305088 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 10:08:16.305173 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66ce7cf41bf942779d67d1d64eac375b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9dd49fc19ab84da6aa61722da0b2afc0"
I20260924 10:08:16.305398 12947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:08:16.306303 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.289231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:45041 (local address 127.0.0.1:55666)
0924 10:08:16.289750 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.289768 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:16.289888 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:16.300225 (+ 10337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:16.300233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:16.300270 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:16.300682 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:16.300694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.302325 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.302329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:16.303672 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:16.303684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.305451 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.305459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:16.305634 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.305675 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 10:08:16.305777 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 10:08:16.305918 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":824,"threads_started":1}
I20260924 10:08:16.306334 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.305950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:60554)
0924 10:08:16.306081 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.306173 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:08:16.306618 13103 leader_election.cc:334] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1930212695dd4fe8b9c68e351946640d (127.6.2.254:41877): Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
I20260924 10:08:16.306953 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.289689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.6.2.252:45041)
0924 10:08:16.299708 (+ 10019us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:16.299716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:16.299746 (+ 30us) server_negotiation.cc:407] Connection header received
0924 10:08:16.299821 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:16.299828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:16.299897 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:16.299998 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:16.300902 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.302177 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.303836 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.305166 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.305319 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.305429 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:16.306089 (+ 660us) server_negotiation.cc:299] Negotiation successful
0924 10:08:16.306225 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9840,"thread_start_us":9629,"threads_started":1}
I20260924 10:08:16.307505 13130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "66ce7cf41bf942779d67d1d64eac375b" is_pre_election: true
I20260924 10:08:16.307847 13130 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dd49fc19ab84da6aa61722da0b2afc0 in current term 2: Already voted for candidate 66ce7cf41bf942779d67d1d64eac375b in this term.
I20260924 10:08:16.308032 12947 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ce7cf41bf942779d67d1d64eac375b in term 2.
I20260924 10:08:16.308373 13106 leader_election.cc:302] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66ce7cf41bf942779d67d1d64eac375b, 9dd49fc19ab84da6aa61722da0b2afc0; no voters: 1930212695dd4fe8b9c68e351946640d
I20260924 10:08:16.308461 12923 leader_election.cc:302] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dd49fc19ab84da6aa61722da0b2afc0; no voters: 1930212695dd4fe8b9c68e351946640d, 66ce7cf41bf942779d67d1d64eac375b
I20260924 10:08:16.308683 13662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:08:16.308807 13666 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:08:16.309226 13662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [term 2 LEADER]: Becoming Leader. State: Replica: 66ce7cf41bf942779d67d1d64eac375b, State: Running, Role: LEADER
I20260924 10:08:16.309600 13662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } }
I20260924 10:08:16.311630 13672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66ce7cf41bf942779d67d1d64eac375b. Latest consensus state: current_term: 2 leader_uuid: "66ce7cf41bf942779d67d1d64eac375b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } } }
I20260924 10:08:16.311923 13672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:16.312505 13674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:08:16.316018 13674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:08:16.316785 13674 catalog_manager.cc:1259] Loaded cluster ID: 3579f76975e44464926a9da06da8f9a6
I20260924 10:08:16.316856 13674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:08:16.317909 13674 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:08:16.318640 13674 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: Loaded TSK: 0
I20260924 10:08:16.319502 13674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:08:16.368156 13120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:08:16.375309 13120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:08:16.379628 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.379281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:41898)
0924 10:08:16.379418 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.379504 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 10:08:16.379887 12947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [term 2 FOLLOWER]: Refusing update from remote peer 66ce7cf41bf942779d67d1d64eac375b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 10:08:16.380716 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.380314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:41908)
0924 10:08:16.380476 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.380541 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 10:08:16.380740 13672 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 10:08:16.381042 13103 consensus_peers.cc:487] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b -> Peer 1930212695dd4fe8b9c68e351946640d (127.6.2.254:41877): Couldn't send request to peer 1930212695dd4fe8b9c68e351946640d. Status: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 10:08:16.383236 13666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66ce7cf41bf942779d67d1d64eac375b. Latest consensus state: current_term: 2 leader_uuid: "66ce7cf41bf942779d67d1d64eac375b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } } }
I20260924 10:08:16.383504 13666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:08:16.384877 13675 mvcc.cc:205] Tried to move back new op lower bound from 7332841457164640256 to 7332841456887341056. Current Snapshot: MvccSnapshot[applied={T|T < 7332841457164640256}]
I20260924 10:08:16.385859 13666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "66ce7cf41bf942779d67d1d64eac375b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } } }
I20260924 10:08:16.386160 13666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dd49fc19ab84da6aa61722da0b2afc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:08:16.387769 13672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "66ce7cf41bf942779d67d1d64eac375b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1930212695dd4fe8b9c68e351946640d" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41877 } } peers { permanent_uuid: "9dd49fc19ab84da6aa61722da0b2afc0" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 38781 } attrs { promote: false } } peers { permanent_uuid: "66ce7cf41bf942779d67d1d64eac375b" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 45041 } attrs { promote: false } } }
I20260924 10:08:16.388070 13672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:16.411849 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.395780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:45265 (local address 127.0.0.1:60786)
0924 10:08:16.396157 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.396175 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:16.396490 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:16.404387 (+ 7897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:16.404391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:16.404418 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:16.404659 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:16.404671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.406656 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.406665 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:16.408957 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:16.408977 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.411261 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.411267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:16.411421 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.411448 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:08:16.411501 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 10:08:16.411599 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 10:08:16.413053 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.403315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.6.2.193:45265)
0924 10:08:16.404040 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:16.404047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:16.404067 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:08:16.404137 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:16.404142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:16.404227 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:16.404799 (+ 572us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:16.404943 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.406451 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.409222 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.411011 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.412409 (+ 1398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.412511 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:16.412567 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 10:08:16.412707 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":247,"threads_started":1}
I20260924 10:08:16.416286 13233 tablet_service.cc:1467] Processing CreateTablet for tablet e6045f583eb24cda87c548c6554a51c3 (DEFAULT_TABLE table=test-table [id=9ac331737b104061853aee30edc6592e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:16.416780 13233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6045f583eb24cda87c548c6554a51c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:16.416992 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.396433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:37877 (local address 127.0.0.1:55572)
0924 10:08:16.396736 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.396753 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:16.396855 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:16.408520 (+ 11665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:16.408527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:16.408578 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:16.409001 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:16.409017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.411554 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.411563 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:16.414416 (+ 2853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:16.414436 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.416291 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.416298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:16.416469 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.416505 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:08:16.416586 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:08:16.416733 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260924 10:08:16.417924 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.396311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.6.2.195:37877)
0924 10:08:16.407928 (+ 11617us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:16.407938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:16.407963 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:08:16.408053 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:16.408061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:16.408162 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:16.409203 (+ 1041us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:16.409414 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.410822 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.414704 (+ 3882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.416021 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.417319 (+ 1298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.417424 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:16.417498 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 10:08:16.417646 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11452,"thread_start_us":12007,"threads_started":1}
I20260924 10:08:16.418340 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.404102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.6.2.194:39965)
0924 10:08:16.411415 (+ 7313us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:16.411424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:16.412194 (+ 770us) server_negotiation.cc:407] Connection header received
0924 10:08:16.412372 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:16.412378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:16.412452 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:16.412962 (+ 510us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:16.413676 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.414583 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.416365 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.417391 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.417546 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.417643 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:16.417927 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 10:08:16.418078 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7135,"thread_start_us":7014,"threads_started":1}
I20260924 10:08:16.418632 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.403594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:39965 (local address 127.0.0.1:34052)
0924 10:08:16.412116 (+ 8522us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.412134 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:16.412243 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:16.412613 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:16.412618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:16.412642 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:16.412827 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:16.412839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.414719 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.414725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:16.416211 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:16.416225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.417545 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.417550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:16.417757 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.417783 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:08:16.417880 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 10:08:16.418184 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8432,"thread_start_us":11764,"threads_started":1}
I20260924 10:08:16.418588 13519 tablet_service.cc:1467] Processing CreateTablet for tablet e6045f583eb24cda87c548c6554a51c3 (DEFAULT_TABLE table=test-table [id=9ac331737b104061853aee30edc6592e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:16.418993 13519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6045f583eb24cda87c548c6554a51c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:16.419404 13378 tablet_service.cc:1467] Processing CreateTablet for tablet e6045f583eb24cda87c548c6554a51c3 (DEFAULT_TABLE table=test-table [id=9ac331737b104061853aee30edc6592e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:16.419860 13378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6045f583eb24cda87c548c6554a51c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:16.423601 13685 tablet_bootstrap.cc:492] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f: Bootstrap starting.
I20260924 10:08:16.424882 13686 tablet_bootstrap.cc:492] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691: Bootstrap starting.
I20260924 10:08:16.425462 13685 tablet_bootstrap.cc:654] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:16.427083 13686 tablet_bootstrap.cc:654] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:16.427882 13685 tablet_bootstrap.cc:492] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f: No bootstrap required, opened a new log
I20260924 10:08:16.427913 13687 tablet_bootstrap.cc:492] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591: Bootstrap starting.
I20260924 10:08:16.428128 13685 ts_tablet_manager.cc:1397] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:08:16.429065 13685 raft_consensus.cc:348] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } } peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } }
I20260924 10:08:16.429329 13685 raft_consensus.cc:374] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:16.429459 13685 raft_consensus.cc:729] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe69c95bbae4afaaf1af62cbb2e918f, State: Initialized, Role: FOLLOWER
I20260924 10:08:16.429636 13687 tablet_bootstrap.cc:654] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:16.429790 13685 consensus_queue.cc:260] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } } peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } }
I20260924 10:08:16.430763 13685 ts_tablet_manager.cc:1428] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:08:16.432126 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.431085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.193:48505)
0924 10:08:16.431713 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.431860 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":785,"threads_started":1}
I20260924 10:08:16.433202 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.432808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.193:37193)
0924 10:08:16.432966 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.433067 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 10:08:16.434332 13299 heartbeater.cc:503] Master 127.6.2.252:45041 was elected leader, sending a full tablet report...
I20260924 10:08:16.436345 13687 tablet_bootstrap.cc:492] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591: No bootstrap required, opened a new log
I20260924 10:08:16.436504 13687 ts_tablet_manager.cc:1397] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260924 10:08:16.437184 13686 tablet_bootstrap.cc:492] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691: No bootstrap required, opened a new log
I20260924 10:08:16.437301 13686 ts_tablet_manager.cc:1397] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260924 10:08:16.437139 13687 raft_consensus.cc:348] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } } peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } }
I20260924 10:08:16.437368 13687 raft_consensus.cc:374] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:16.437480 13687 raft_consensus.cc:729] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7258ad39bc4a4b29ab53cd2483c8c591, State: Initialized, Role: FOLLOWER
I20260924 10:08:16.437747 13687 consensus_queue.cc:260] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } } peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } }
I20260924 10:08:16.437883 13686 raft_consensus.cc:348] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } } peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } }
I20260924 10:08:16.438054 13686 raft_consensus.cc:374] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:16.438122 13686 raft_consensus.cc:729] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356ebd95dd6a477ea2216bf6bc20e691, State: Initialized, Role: FOLLOWER
I20260924 10:08:16.438334 13686 consensus_queue.cc:260] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } } peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } }
I20260924 10:08:16.438607 13687 ts_tablet_manager.cc:1428] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:08:16.439572 13686 ts_tablet_manager.cc:1428] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 10:08:16.441893 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.438923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.195:60409)
0924 10:08:16.439496 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.439626 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":307,"threads_started":1}
I20260924 10:08:16.442499 13444 heartbeater.cc:503] Master 127.6.2.252:45041 was elected leader, sending a full tablet report...
I20260924 10:08:16.442657 13585 heartbeater.cc:503] Master 127.6.2.252:45041 was elected leader, sending a full tablet report...
I20260924 10:08:16.443007 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.439790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.194:53823)
0924 10:08:16.442460 (+ 2670us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.442669 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2546,"thread_start_us":2459,"threads_started":1}
I20260924 10:08:16.443990 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.443646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.194:51149)
0924 10:08:16.443780 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.443863 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 10:08:16.444747 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.444387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.195:58137)
0924 10:08:16.444506 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.444617 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 10:08:16.604986 13694 raft_consensus.cc:482] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:08:16.605269 13694 raft_consensus.cc:504] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } } peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } }
I20260924 10:08:16.606061 13694 leader_election.cc:288] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbe69c95bbae4afaaf1af62cbb2e918f (127.6.2.193:45265), 7258ad39bc4a4b29ab53cd2483c8c591 (127.6.2.195:37877)
I20260924 10:08:16.606674 13539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6045f583eb24cda87c548c6554a51c3" candidate_uuid: "356ebd95dd6a477ea2216bf6bc20e691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" is_pre_election: true
I20260924 10:08:16.606729 13253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6045f583eb24cda87c548c6554a51c3" candidate_uuid: "356ebd95dd6a477ea2216bf6bc20e691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" is_pre_election: true
I20260924 10:08:16.606938 13539 raft_consensus.cc:2454] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 356ebd95dd6a477ea2216bf6bc20e691 in term 0.
I20260924 10:08:16.606992 13253 raft_consensus.cc:2454] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 356ebd95dd6a477ea2216bf6bc20e691 in term 0.
I20260924 10:08:16.607512 13333 leader_election.cc:302] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 356ebd95dd6a477ea2216bf6bc20e691, 7258ad39bc4a4b29ab53cd2483c8c591; no voters:
I20260924 10:08:16.607867 13694 raft_consensus.cc:2781] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:08:16.607988 13694 raft_consensus.cc:482] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:08:16.608060 13694 raft_consensus.cc:3037] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:16.611987 13694 raft_consensus.cc:504] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } } peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } }
I20260924 10:08:16.612553 13694 leader_election.cc:288] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [CANDIDATE]: Term 1 election: Requested vote from peers cbe69c95bbae4afaaf1af62cbb2e918f (127.6.2.193:45265), 7258ad39bc4a4b29ab53cd2483c8c591 (127.6.2.195:37877)
I20260924 10:08:16.613072 13253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6045f583eb24cda87c548c6554a51c3" candidate_uuid: "356ebd95dd6a477ea2216bf6bc20e691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f"
I20260924 10:08:16.613109 13539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6045f583eb24cda87c548c6554a51c3" candidate_uuid: "356ebd95dd6a477ea2216bf6bc20e691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7258ad39bc4a4b29ab53cd2483c8c591"
I20260924 10:08:16.613286 13253 raft_consensus.cc:3037] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:16.613333 13539 raft_consensus.cc:3037] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:16.615293 13253 raft_consensus.cc:2454] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 356ebd95dd6a477ea2216bf6bc20e691 in term 1.
I20260924 10:08:16.615350 13539 raft_consensus.cc:2454] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 356ebd95dd6a477ea2216bf6bc20e691 in term 1.
I20260924 10:08:16.615770 13332 leader_election.cc:302] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 356ebd95dd6a477ea2216bf6bc20e691, cbe69c95bbae4afaaf1af62cbb2e918f; no voters:
I20260924 10:08:16.616097 13694 raft_consensus.cc:2781] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:16.616539 13694 raft_consensus.cc:686] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [term 1 LEADER]: Becoming Leader. State: Replica: 356ebd95dd6a477ea2216bf6bc20e691, State: Running, Role: LEADER
I20260924 10:08:16.616847 13694 consensus_queue.cc:237] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } } peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } }
I20260924 10:08:16.617645 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.617136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.194:59297)
0924 10:08:16.617323 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.617449 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 10:08:16.619794 13120 catalog_manager.cc:5495] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 reported cstate change: term changed from 0 to 1, leader changed from <none> to 356ebd95dd6a477ea2216bf6bc20e691 (127.6.2.194). New cstate: current_term: 1 leader_uuid: "356ebd95dd6a477ea2216bf6bc20e691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "356ebd95dd6a477ea2216bf6bc20e691" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 39965 } health_report { overall_health: HEALTHY } } }
I20260924 10:08:16.643939 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.639220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34060 (local address 127.6.2.194:39965)
0924 10:08:16.639411 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:16.639419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:16.639451 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:08:16.639709 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:16.639714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:16.639774 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:16.639861 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:08:16.640253 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.640944 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.642296 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.642723 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.642859 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.643268 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:08:16.643383 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:08:16.643483 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:16.643626 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 10:08:16.643700 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:08:16.646939 13253 raft_consensus.cc:1261] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f [term 1 FOLLOWER]: Refusing update from remote peer 356ebd95dd6a477ea2216bf6bc20e691: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:16.646941 13539 raft_consensus.cc:1261] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 1 FOLLOWER]: Refusing update from remote peer 356ebd95dd6a477ea2216bf6bc20e691: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:16.647610 13697 consensus_queue.cc:1035] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7258ad39bc4a4b29ab53cd2483c8c591" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 37877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:16.647941 13694 consensus_queue.cc:1035] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbe69c95bbae4afaaf1af62cbb2e918f" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:16.650074 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.649456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.193:49115)
0924 10:08:16.649736 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.649862 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260924 10:08:16.650595 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.650068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.195:52717)
0924 10:08:16.650296 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.650415 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260924 10:08:16.656905 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.656126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.193:45995)
0924 10:08:16.656650 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.656757 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408}
I20260924 10:08:16.657400 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.656251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.194:55273)
0924 10:08:16.657154 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.657238 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":800}
I20260924 10:08:16.660743 13701 mvcc.cc:205] Tried to move back new op lower bound from 7332841458260312064 to 7332841458144800768. Current Snapshot: MvccSnapshot[applied={T|T < 7332841458260312064}]
I20260924 10:08:16.664453 13120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:16.664767 13120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:16.668514 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.667874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.194:39539)
0924 10:08:16.668236 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.668332 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240}
I20260924 10:08:16.671180 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.670645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.6.2.195:39983)
0924 10:08:16.670869 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.670986 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 10:08:16.673480 13120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66ce7cf41bf942779d67d1d64eac375b: Sending DeleteTablet for 3 replicas of tablet e6045f583eb24cda87c548c6554a51c3
I20260924 10:08:16.674726 13519 tablet_service.cc:1514] Processing DeleteTablet for tablet e6045f583eb24cda87c548c6554a51c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:16 UTC) from {username='oryx', principal='oryx/127.6.2.252@KRBTEST.COM'} at 127.0.0.1:55572
I20260924 10:08:16.675204 13518 tablet_service.cc:1514] Processing DeleteTablet for tablet e6045f583eb24cda87c548c6554a51c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:16 UTC) from {username='oryx', principal='oryx/127.6.2.252@KRBTEST.COM'} at 127.0.0.1:55572
I20260924 10:08:16.675227 13378 tablet_service.cc:1514] Processing DeleteTablet for tablet e6045f583eb24cda87c548c6554a51c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:16 UTC) from {username='oryx', principal='oryx/127.6.2.252@KRBTEST.COM'} at 127.0.0.1:34052
I20260924 10:08:16.675390 13233 tablet_service.cc:1514] Processing DeleteTablet for tablet e6045f583eb24cda87c548c6554a51c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:16 UTC) from {username='oryx', principal='oryx/127.6.2.252@KRBTEST.COM'} at 127.0.0.1:60786
I20260924 10:08:16.675249 13377 tablet_service.cc:1514] Processing DeleteTablet for tablet e6045f583eb24cda87c548c6554a51c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:16 UTC) from {username='oryx', principal='oryx/127.6.2.252@KRBTEST.COM'} at 127.0.0.1:34052
I20260924 10:08:16.677554 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 13107
I20260924 10:08:16.685330 13710 tablet_replica.cc:330] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f: stopping tablet replica
I20260924 10:08:16.685535 13710 raft_consensus.cc:2229] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:16.685853 13710 raft_consensus.cc:2258] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:16.687492 13710 ts_tablet_manager.cc:1905] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:16.688814 13708 tablet_replica.cc:330] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591: stopping tablet replica
I20260924 10:08:16.689363 13709 tablet_replica.cc:330] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691: stopping tablet replica
I20260924 10:08:16.689549 13709 raft_consensus.cc:2229] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:08:16.689956 13709 raft_consensus.cc:2258] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:16.690320 13708 raft_consensus.cc:2229] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:16.690622 13708 raft_consensus.cc:2258] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:16.691185 13709 ts_tablet_manager.cc:1905] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:16.691569 13708 ts_tablet_manager.cc:1905] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:16.695375 13710 ts_tablet_manager.cc:1918] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:16.695564 13710 log.cc:1196] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/e6045f583eb24cda87c548c6554a51c3
I20260924 10:08:16.696094 13710 ts_tablet_manager.cc:1939] T e6045f583eb24cda87c548c6554a51c3 P cbe69c95bbae4afaaf1af62cbb2e918f: Deleting consensus metadata
I20260924 10:08:16.697320 13104 catalog_manager.cc:4841] TS cbe69c95bbae4afaaf1af62cbb2e918f (127.6.2.193:45265): tablet e6045f583eb24cda87c548c6554a51c3 (table test-table [id=9ac331737b104061853aee30edc6592e]) successfully deleted
I20260924 10:08:16.697883 13708 ts_tablet_manager.cc:1918] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:16.697989 13709 ts_tablet_manager.cc:1918] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:16.698031 13708 log.cc:1196] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/e6045f583eb24cda87c548c6554a51c3
I20260924 10:08:16.698129 13709 log.cc:1196] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/e6045f583eb24cda87c548c6554a51c3
I20260924 10:08:16.698436 13708 ts_tablet_manager.cc:1939] T e6045f583eb24cda87c548c6554a51c3 P 7258ad39bc4a4b29ab53cd2483c8c591: Deleting consensus metadata
I20260924 10:08:16.698601 13709 ts_tablet_manager.cc:1939] T e6045f583eb24cda87c548c6554a51c3 P 356ebd95dd6a477ea2216bf6bc20e691: Deleting consensus metadata
I20260924 10:08:16.699463 13105 catalog_manager.cc:4841] TS 7258ad39bc4a4b29ab53cd2483c8c591 (127.6.2.195:37877): tablet e6045f583eb24cda87c548c6554a51c3 (table test-table [id=9ac331737b104061853aee30edc6592e]) successfully deleted
I20260924 10:08:16.699743 13104 catalog_manager.cc:4841] TS 356ebd95dd6a477ea2216bf6bc20e691 (127.6.2.194:39965): tablet e6045f583eb24cda87c548c6554a51c3 (table test-table [id=9ac331737b104061853aee30edc6592e]) successfully deleted
W20260924 10:08:16.699939 13105 catalog_manager.cc:4808] TS 7258ad39bc4a4b29ab53cd2483c8c591 (127.6.2.195:37877): delete failed for tablet e6045f583eb24cda87c548c6554a51c3 because the tablet was not found. No further retry: Not found: Tablet not found: e6045f583eb24cda87c548c6554a51c3
W20260924 10:08:16.700523 13104 catalog_manager.cc:4808] TS 356ebd95dd6a477ea2216bf6bc20e691 (127.6.2.194:39965): delete failed for tablet e6045f583eb24cda87c548c6554a51c3 because the tablet was not found. No further retry: Not found: Tablet not found: e6045f583eb24cda87c548c6554a51c3
I20260924 10:08:16.704142 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.678853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.6.2.193:45265)
0924 10:08:16.679111 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:16.679121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:16.688397 (+ 9276us) server_negotiation.cc:407] Connection header received
0924 10:08:16.688798 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:16.688806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:16.688911 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:16.689046 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:16.690895 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.691990 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.693746 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.694418 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.694516 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.698054 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:16.698074 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:16.698097 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:16.698140 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:16.701666 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:16.702285 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:16.702292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:16.702295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:16.702408 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:16.702744 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:16.702747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:16.702749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:16.703213 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:16.703434 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:16.703756 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 10:08:16.703910 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:08:16.704720 13293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:16.798838 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 13107
I20260924 10:08:16.816394 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 13310
I20260924 10:08:16.829803 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.817222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34070 (local address 127.6.2.194:39965)
0924 10:08:16.817416 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:16.817424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:16.817460 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:08:16.817687 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:16.817691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:16.817762 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:16.817866 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:16.819140 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.819863 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.821054 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.821444 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.821576 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.824188 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:16.824203 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:16.824219 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:16.824249 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:16.827007 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:16.827787 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:16.827801 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:16.827804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:16.827930 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:16.828295 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:16.828300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:16.828303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:16.828659 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:16.828954 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:16.829251 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 10:08:16.829463 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 10:08:16.830487 13438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:16.934784 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.934096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:41924)
0924 10:08:16.934521 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.934612 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":235,"threads_started":1}
I20260924 10:08:16.936089 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.935775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41877 (local address 127.0.0.1:41938)
0924 10:08:16.935908 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:16.935979 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 10:08:16.957417 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 13310
I20260924 10:08:16.980088 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 13456
I20260924 10:08:16.994369 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:16.981002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55582 (local address 127.6.2.195:37877)
0924 10:08:16.981446 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:16.981450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:16.981466 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:08:16.981519 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:16.981523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:16.981593 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:16.981721 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:16.983232 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.984018 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.985260 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:16.985724 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:16.985849 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:16.988527 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:16.988546 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:16.988570 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:16.988608 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:16.991545 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:16.992217 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:16.992231 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:16.992234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:16.992364 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:16.992740 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:16.992756 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:16.992759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:16.993146 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:16.993477 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:16.993812 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 10:08:16.994049 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":249,"threads_started":1}
I20260924 10:08:16.995399 13579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:17.089619 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 13456
I20260924 10:08:17.106415 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 12793
I20260924 10:08:17.107540 12967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:17.233366 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 12793
I20260924 10:08:17.259801 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 12892
I20260924 10:08:17.260979 13150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:17.402335 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 12892
2026-09-24T10:08:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9997 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:08:17 dist-test-slave-s3n3 krb5kdc[13722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:08:17 dist-test-slave-s3n3 krb5kdc[13722](info): set up 2 sockets
Sep 24 10:08:17 dist-test-slave-s3n3 krb5kdc[13722](info): commencing operation
krb5kdc: starting...
W20260924 10:08:18.540174 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:08:18 dist-test-slave-s3n3 krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:08:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:08:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.254.keytab.
Entry for principal oryx/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.254.keytab.
I20260924 10:08:18.763506 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:46055
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.254.keytab
--principal=oryx/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:46055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:19.085013 13735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:19.085346 13735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:19.085496 13735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:19.092808 13735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:19.092936 13735 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:08:19.093029 13735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:19.093140 13735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:19.093221 13735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:19.093312 13735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:08:19.103164 13735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:46055
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.254.keytab
--principal=oryx/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:46055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:19.104410 13735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:19.105724 13735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:19.117093 13740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:19.117312 13741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:19.117601 13743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:19.118054 13735 server_base.cc:1034] running on GCE node
I20260924 10:08:19.118767 13735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:19.120131 13735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:19.121371 13735 hybrid_clock.cc:648] HybridClock initialized: now 1790244499121312 us; error 40 us; skew 500 ppm
I20260924 10:08:19.121738 13735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:19 dist-test-slave-s3n3 krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244499, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:19.124794 13735 init.cc:377] Logged in from keytab as oryx/127.6.2.254@KRBTEST.COM (short username oryx)
I20260924 10:08:19.126164 13735 webserver.cc:460] Webserver started at http://127.6.2.254:45523/ using document root <none> and password file <none>
I20260924 10:08:19.126806 13735 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:19.126888 13735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:19.127231 13735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:19.130160 13735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "32c9b81de2f14df98bbc6eb56f282e0b"
format_stamp: "Formatted at 2026-09-24 10:08:19 on dist-test-slave-s3n3"
server_key: "1871692ac11bf0bd0856b2c86b0b017d"
server_key_iv: "48f93c123cc9711911af1e9bfb9e8a57"
server_key_version: "encryptionkey@0"
I20260924 10:08:19.130901 13735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "32c9b81de2f14df98bbc6eb56f282e0b"
format_stamp: "Formatted at 2026-09-24 10:08:19 on dist-test-slave-s3n3"
server_key: "1871692ac11bf0bd0856b2c86b0b017d"
server_key_iv: "48f93c123cc9711911af1e9bfb9e8a57"
server_key_version: "encryptionkey@0"
I20260924 10:08:19.136053 13735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:08:19.139402 13750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:19.140671 13735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:08:19.140847 13735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "32c9b81de2f14df98bbc6eb56f282e0b"
format_stamp: "Formatted at 2026-09-24 10:08:19 on dist-test-slave-s3n3"
server_key: "1871692ac11bf0bd0856b2c86b0b017d"
server_key_iv: "48f93c123cc9711911af1e9bfb9e8a57"
server_key_version: "encryptionkey@0"
I20260924 10:08:19.141053 13735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:19.167805 13735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:19.172245 13735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:19.172537 13735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:19.193279 13735 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:46055
I20260924 10:08:19.193300 13811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:46055 every 8 connection(s)
I20260924 10:08:19.194706 13735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:08:19.199092 13812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:19.200203 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 13735
I20260924 10:08:19.200412 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:08:19.201228 6155 external_mini_cluster.cc:1428] Setting key 325b4300eb31da97227c98e241212b57
I20260924 10:08:19.213567 13812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b: Bootstrap starting.
Sep 24 10:08:19 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:19.217078 13812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:19.218272 13812 log.cc:824] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:19.220975 13812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b: No bootstrap required, opened a new log
I20260924 10:08:19.223752 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:19.203397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38832 (local address 127.6.2.254:46055)
0924 10:08:19.204261 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:19.204273 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:19.204313 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:08:19.205890 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:19.205911 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:19.206299 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:19.206618 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:19.208440 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:19.209729 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:19.211513 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:19.212202 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:19.212328 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:19.216728 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:19.216736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:19.216750 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:19.216789 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:19.219841 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:19.220541 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:19.220547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:19.220553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:19.220715 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:19.221078 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:19.221083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:19.221086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:19.221567 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:19.221867 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:19.222265 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 10:08:19.222694 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":383,"threads_started":1}
I20260924 10:08:19.228790 13812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c9b81de2f14df98bbc6eb56f282e0b" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46055 } }
I20260924 10:08:19.229122 13812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:19.229219 13812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c9b81de2f14df98bbc6eb56f282e0b, State: Initialized, Role: FOLLOWER
I20260924 10:08:19.229712 13812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c9b81de2f14df98bbc6eb56f282e0b" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46055 } }
I20260924 10:08:19.229944 13812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:08:19.230078 13812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:08:19.230227 13812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:19.232307 13812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c9b81de2f14df98bbc6eb56f282e0b" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46055 } }
I20260924 10:08:19.232798 13812 leader_election.cc:302] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32c9b81de2f14df98bbc6eb56f282e0b; no voters:
I20260924 10:08:19.233325 13812 leader_election.cc:288] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:08:19.233464 13817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:19.234755 13817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [term 1 LEADER]: Becoming Leader. State: Replica: 32c9b81de2f14df98bbc6eb56f282e0b, State: Running, Role: LEADER
I20260924 10:08:19.235378 13817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c9b81de2f14df98bbc6eb56f282e0b" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46055 } }
I20260924 10:08:19.235710 13812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:08:19.238245 13819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c9b81de2f14df98bbc6eb56f282e0b. Latest consensus state: current_term: 1 leader_uuid: "32c9b81de2f14df98bbc6eb56f282e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c9b81de2f14df98bbc6eb56f282e0b" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46055 } } }
I20260924 10:08:19.238343 13818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32c9b81de2f14df98bbc6eb56f282e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c9b81de2f14df98bbc6eb56f282e0b" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46055 } } }
I20260924 10:08:19.238689 13819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:19.238718 13818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:19.240531 13822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:08:19.246070 13822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:08:19.255502 13822 catalog_manager.cc:1347] Generated new cluster ID: 8301f2e636954ffdb75afc559dccf79a
I20260924 10:08:19.255663 13822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:08:19.275661 13822 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:08:19.277006 13822 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:08:19.284155 13822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b: Generated new TSK 0
I20260924 10:08:19.284947 13822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab.
Entry for principal oryx/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab.
I20260924 10:08:19.347007 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46055
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab
--principal=oryx/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:19.677053 13838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:19.677361 13838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:19.677487 13838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:19.684799 13838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:19.684919 13838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:19.685006 13838 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:08:19.685086 13838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:08:19.694891 13838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab
--principal=oryx/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.2.254:46055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:19.696230 13838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:19.697520 13838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:19.710431 13843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:19.710455 13844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:19.710393 13846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:19.711146 13838 server_base.cc:1034] running on GCE node
I20260924 10:08:19.711761 13838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:19.712773 13838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:19.714004 13838 hybrid_clock.cc:648] HybridClock initialized: now 1790244499713927 us; error 56 us; skew 500 ppm
I20260924 10:08:19.714262 13838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:19 dist-test-slave-s3n3 krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244499, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:19.717332 13838 init.cc:377] Logged in from keytab as oryx/127.6.2.193@KRBTEST.COM (short username oryx)
I20260924 10:08:19.718721 13838 webserver.cc:460] Webserver started at http://127.6.2.193:41249/ using document root <none> and password file <none>
I20260924 10:08:19.719489 13838 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:19.719597 13838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:19.719924 13838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:19.722718 13838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "f86812a9280f479bbcce24732391b5c5"
format_stamp: "Formatted at 2026-09-24 10:08:19 on dist-test-slave-s3n3"
server_key: "94ae44400cbd5a13e2ef426aa4073d32"
server_key_iv: "5622dc615cf6111447fe717835c6cad6"
server_key_version: "encryptionkey@0"
I20260924 10:08:19.723511 13838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "f86812a9280f479bbcce24732391b5c5"
format_stamp: "Formatted at 2026-09-24 10:08:19 on dist-test-slave-s3n3"
server_key: "94ae44400cbd5a13e2ef426aa4073d32"
server_key_iv: "5622dc615cf6111447fe717835c6cad6"
server_key_version: "encryptionkey@0"
I20260924 10:08:19.729209 13838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:08:19.732720 13853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:19.734107 13838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:08:19.734421 13838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "f86812a9280f479bbcce24732391b5c5"
format_stamp: "Formatted at 2026-09-24 10:08:19 on dist-test-slave-s3n3"
server_key: "94ae44400cbd5a13e2ef426aa4073d32"
server_key_iv: "5622dc615cf6111447fe717835c6cad6"
server_key_version: "encryptionkey@0"
I20260924 10:08:19.734673 13838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:19.752185 13838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:19.756731 13838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:19.757057 13838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:19.759907 13838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:19.760011 13838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:19.760165 13838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:19.760253 13838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:08:19 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244499, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:19.785256 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:19.760373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:49667 (local address 127.6.2.254:46055)
0924 10:08:19.761074 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:19.761081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:19.762115 (+ 1034us) server_negotiation.cc:407] Connection header received
0924 10:08:19.764175 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:19.764185 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:19.764289 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:19.764421 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:19.766588 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:19.767968 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:19.770437 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:19.771252 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:19.771403 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:19.777122 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:19.777131 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:19.777136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:19.777186 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:19.780888 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:19.782137 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:19.782145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:19.782150 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:19.782317 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:19.783102 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:19.783108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:19.783112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:19.783494 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:19.783651 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:19.784616 (+ 965us) server_negotiation.cc:299] Negotiation successful
0924 10:08:19.784851 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":478,"threads_started":1}
I20260924 10:08:19.786844 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:19.761014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:46055 (local address 127.6.2.193:49667)
0924 10:08:19.761935 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:19.761984 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:19.763889 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:19.764672 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:19.764679 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:19.765468 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:19.766351 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:19.766383 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:19.768147 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:19.768152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:19.770314 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:19.770326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:19.771561 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:19.771574 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:19.771926 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:19.772875 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:19.772903 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:19.776798 (+ 3895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:19.781072 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:19.781087 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:19.781107 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:19.781905 (+ 798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:19.782454 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:19.782460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:19.782464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:19.782862 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:19.783642 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:19.783651 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:19.784060 (+ 409us) client_negotiation.cc:769] Sending connection context
0924 10:08:19.784533 (+ 473us) client_negotiation.cc:240] Negotiation successful
0924 10:08:19.785148 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":398,"threads_started":1}
I20260924 10:08:19.807956 13838 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:41261
I20260924 10:08:19.808156 13973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:41261 every 8 connection(s)
I20260924 10:08:19.809056 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:19.793658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:56175 (local address 127.6.2.254:46055)
0924 10:08:19.793834 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:19.793839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:19.794547 (+ 708us) server_negotiation.cc:407] Connection header received
0924 10:08:19.794776 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:19.794781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:19.794879 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:19.794982 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:19.796502 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:19.797200 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:19.799697 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:19.800209 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:19.800366 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:19.803122 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:19.803128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:19.803137 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:19.803168 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:19.806281 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:19.806965 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:19.806973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:19.806976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:19.807192 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:19.807713 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:19.807720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:19.807725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:19.808104 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:19.808266 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:19.808682 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 10:08:19.808842 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:08:19.809384 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:19.793751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:46055 (local address 127.6.2.193:56175)
0924 10:08:19.794410 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:19.794425 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:19.794626 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:19.795102 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:19.795110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:19.795838 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:19.796317 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:19.796333 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:19.797391 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:19.797414 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:19.799491 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:19.799512 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:19.800479 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:19.800489 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:19.800780 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:19.801621 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:19.801641 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:19.802916 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:19.806415 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:19.806421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:19.806425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:19.806839 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:19.807369 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:19.807378 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:19.807388 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:19.807573 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:19.808229 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:19.808234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:19.808433 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 10:08:19.808592 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 10:08:19.808913 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":368,"threads_started":1}
I20260924 10:08:19.809616 13838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:08:19.811880 13974 heartbeater.cc:347] Connected to a master server at 127.6.2.254:46055
I20260924 10:08:19.812511 13974 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:19.813344 13974 heartbeater.cc:511] Master 127.6.2.254:46055 requested a full tablet report, sending...
I20260924 10:08:19.815322 13767 ts_manager.cc:194] Registered new tserver with Master: f86812a9280f479bbcce24732391b5c5 (127.6.2.193:41261)
I20260924 10:08:19.816690 13767 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:49667
I20260924 10:08:19.818670 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 13838
I20260924 10:08:19.818846 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:08:19.819510 6155 external_mini_cluster.cc:1428] Setting key be846e6a26977039c8c568408e2d1718
WARNING: no policy specified for oryx/127.6.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab.
Entry for principal oryx/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab.
I20260924 10:08:19.867344 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46055
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab
--principal=oryx/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:20.201532 13978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:20.201854 13978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:20.202018 13978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:20.209455 13978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:20.209607 13978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:20.209774 13978 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:08:20.209919 13978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:08:20.220312 13978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab
--principal=oryx/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.2.254:46055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:20.221542 13978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:20.222806 13978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:20.236034 13983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:20.236009 13986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:20.235970 13984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:20.236883 13978 server_base.cc:1034] running on GCE node
I20260924 10:08:20.237519 13978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:20.238476 13978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:20.239734 13978 hybrid_clock.cc:648] HybridClock initialized: now 1790244500239674 us; error 40 us; skew 500 ppm
I20260924 10:08:20.240072 13978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:20 dist-test-slave-s3n3 krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244500, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:20.243933 13978 init.cc:377] Logged in from keytab as oryx/127.6.2.194@KRBTEST.COM (short username oryx)
I20260924 10:08:20.245744 13978 webserver.cc:460] Webserver started at http://127.6.2.194:38515/ using document root <none> and password file <none>
I20260924 10:08:20.246532 13978 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:20.246667 13978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:20.247014 13978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:20.250818 13978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "ca4d5fa0bfa64ca0a5254009350e3044"
format_stamp: "Formatted at 2026-09-24 10:08:20 on dist-test-slave-s3n3"
server_key: "2511d4d7e92bb4385b8de99d016ba9b4"
server_key_iv: "6cf50e694cb9e30b4f2a26e312520d10"
server_key_version: "encryptionkey@0"
I20260924 10:08:20.251802 13978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "ca4d5fa0bfa64ca0a5254009350e3044"
format_stamp: "Formatted at 2026-09-24 10:08:20 on dist-test-slave-s3n3"
server_key: "2511d4d7e92bb4385b8de99d016ba9b4"
server_key_iv: "6cf50e694cb9e30b4f2a26e312520d10"
server_key_version: "encryptionkey@0"
I20260924 10:08:20.257709 13978 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 10:08:20.261286 13993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:20.262691 13978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:08:20.262889 13978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "ca4d5fa0bfa64ca0a5254009350e3044"
format_stamp: "Formatted at 2026-09-24 10:08:20 on dist-test-slave-s3n3"
server_key: "2511d4d7e92bb4385b8de99d016ba9b4"
server_key_iv: "6cf50e694cb9e30b4f2a26e312520d10"
server_key_version: "encryptionkey@0"
I20260924 10:08:20.263142 13978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:20.297614 13978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:20.301961 13978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:20.302280 13978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:20.305244 13978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:20.305436 13978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 10:08:20.305667 13978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:20.305794 13978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:08:20 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244500, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:20.332499 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.305763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:54337 (local address 127.6.2.254:46055)
0924 10:08:20.305927 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:20.305932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:20.307607 (+ 1675us) server_negotiation.cc:407] Connection header received
0924 10:08:20.309657 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:20.309663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:20.309754 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:20.309864 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:20.312202 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.313349 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.316585 (+ 3236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.317414 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.317600 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:20.322972 (+ 5372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:20.322982 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:20.322989 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:20.323037 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:20.326972 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:20.328558 (+ 1586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:20.328565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:20.328569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:20.328741 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:20.329352 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:20.329361 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:20.329367 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:20.329886 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:20.330107 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:20.331557 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0924 10:08:20.331912 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:08:20.335098 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.306529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:46055 (local address 127.6.2.194:54337)
0924 10:08:20.307431 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:20.307474 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:20.309371 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:20.310082 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:20.310090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:20.310868 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:20.311978 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:20.312005 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.314119 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.314126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:20.316404 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:20.316427 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.317734 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.317746 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:20.318022 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:20.318828 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:20.318874 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:20.322644 (+ 3770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:20.327366 (+ 4722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:20.327420 (+ 54us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:20.327453 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:20.328386 (+ 933us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:20.328881 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:20.328891 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:20.328895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:20.329144 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:20.330144 (+ 1000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:20.330154 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:20.330837 (+ 683us) client_negotiation.cc:769] Sending connection context
0924 10:08:20.331376 (+ 539us) client_negotiation.cc:240] Negotiation successful
0924 10:08:20.332286 (+ 910us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":384,"threads_started":1}
I20260924 10:08:20.359644 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.343708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:46055 (local address 127.6.2.194:40913)
0924 10:08:20.344561 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:20.344577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:20.344761 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:20.345112 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:20.345117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:20.345683 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:20.346033 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:20.346048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.347783 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.347789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:20.348814 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:20.348825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.350068 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.350073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:20.350204 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:20.350923 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:20.350940 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:20.352513 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:20.356399 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:20.356405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:20.356410 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:20.356973 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:20.357379 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:20.357383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:20.357387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:20.357549 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:20.358322 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:20.358326 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:20.358615 (+ 289us) client_negotiation.cc:769] Sending connection context
0924 10:08:20.358803 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 10:08:20.359173 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":621,"threads_started":1}
I20260924 10:08:20.359807 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.344272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:40913 (local address 127.6.2.254:46055)
0924 10:08:20.344488 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:20.344495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:20.344675 (+ 180us) server_negotiation.cc:407] Connection header received
0924 10:08:20.344918 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:20.344923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:20.344985 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:20.345088 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:20.346247 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.347609 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.348962 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.349878 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.350039 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:20.352729 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:20.352737 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:20.352744 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:20.352793 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:20.356270 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:20.357117 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:20.357124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:20.357128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:20.357259 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:20.357663 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:20.357670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:20.357674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:20.358109 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:20.358260 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:20.359245 (+ 985us) server_negotiation.cc:299] Negotiation successful
0924 10:08:20.359472 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:08:20.368621 13978 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:44605
I20260924 10:08:20.368651 14112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:44605 every 8 connection(s)
I20260924 10:08:20.369778 13978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:08:20.370241 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 13978
I20260924 10:08:20.370406 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:08:20.370862 6155 external_mini_cluster.cc:1428] Setting key 0f3bfefdc3019e1271a7c3b72b41839e
I20260924 10:08:20.372139 14113 heartbeater.cc:347] Connected to a master server at 127.6.2.254:46055
I20260924 10:08:20.372437 14113 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:20.373450 14113 heartbeater.cc:511] Master 127.6.2.254:46055 requested a full tablet report, sending...
I20260924 10:08:20.375375 13767 ts_manager.cc:194] Registered new tserver with Master: ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194:44605)
I20260924 10:08:20.376255 13767 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:54337
WARNING: no policy specified for oryx/127.6.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab.
Entry for principal oryx/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab.
I20260924 10:08:20.419746 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46055
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab
--principal=oryx/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:20.755167 14117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:20.755514 14117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:20.755733 14117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:20.763995 14117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:20.764164 14117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:20.764312 14117 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:08:20.764451 14117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:08:20.775483 14117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab
--principal=oryx/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.2.254:46055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:20.776777 14117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:20.778124 14117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:20.791640 14123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:20.791692 14122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:20.791822 14125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:20.792503 14117 server_base.cc:1034] running on GCE node
I20260924 10:08:20.793159 14117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:20.794116 14117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:20.795374 14117 hybrid_clock.cc:648] HybridClock initialized: now 1790244500795308 us; error 45 us; skew 500 ppm
I20260924 10:08:20.795691 14117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:20 dist-test-slave-s3n3 krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244500, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:20.798907 14117 init.cc:377] Logged in from keytab as oryx/127.6.2.195@KRBTEST.COM (short username oryx)
I20260924 10:08:20.800498 14117 webserver.cc:460] Webserver started at http://127.6.2.195:39561/ using document root <none> and password file <none>
I20260924 10:08:20.801234 14117 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:20.801369 14117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:20.801699 14117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:20.804895 14117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "6c8d7c65d75b443c81a6a512ded1d1b1"
format_stamp: "Formatted at 2026-09-24 10:08:20 on dist-test-slave-s3n3"
server_key: "1c872b5ba23f4f36615a7b424a463c66"
server_key_iv: "1698956b999852dedf87eb2ea2aa6876"
server_key_version: "encryptionkey@0"
I20260924 10:08:20.805774 14117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "6c8d7c65d75b443c81a6a512ded1d1b1"
format_stamp: "Formatted at 2026-09-24 10:08:20 on dist-test-slave-s3n3"
server_key: "1c872b5ba23f4f36615a7b424a463c66"
server_key_iv: "1698956b999852dedf87eb2ea2aa6876"
server_key_version: "encryptionkey@0"
I20260924 10:08:20.811430 14117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 10:08:20.815140 14132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:20.816531 14117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:08:20.816716 14117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "6c8d7c65d75b443c81a6a512ded1d1b1"
format_stamp: "Formatted at 2026-09-24 10:08:20 on dist-test-slave-s3n3"
server_key: "1c872b5ba23f4f36615a7b424a463c66"
server_key_iv: "1698956b999852dedf87eb2ea2aa6876"
server_key_version: "encryptionkey@0"
I20260924 10:08:20.816963 14117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:20.819435 13974 heartbeater.cc:503] Master 127.6.2.254:46055 was elected leader, sending a full tablet report...
I20260924 10:08:20.839912 14117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:20.844131 14117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:20.844439 14117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:20.847460 14117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:20.847539 14117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:20.847711 14117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:20.847764 14117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:08:20 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244500, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:20.873803 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.848023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:53633 (local address 127.6.2.254:46055)
0924 10:08:20.848220 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:20.848227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:20.849774 (+ 1547us) server_negotiation.cc:407] Connection header received
0924 10:08:20.852039 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:20.852044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:20.852266 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:20.852406 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:20.856071 (+ 3665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.856792 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.860254 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.860730 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.860834 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:20.866301 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:20.866310 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:20.866317 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:20.866369 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:20.869884 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:20.870981 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:20.870986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:20.870990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:20.871216 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:20.871882 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:20.871886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:20.871888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:20.872164 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:20.872282 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:20.873368 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0924 10:08:20.873542 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:08:20.876085 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.848675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:46055 (local address 127.6.2.195:53633)
0924 10:08:20.849581 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:20.849636 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:20.851747 (+ 2111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:20.853101 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:20.853119 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:20.854672 (+ 1553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:20.855810 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:20.855851 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.856969 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.856976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:20.860005 (+ 3029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:20.860024 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.860993 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.861003 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:20.861373 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:20.862399 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:20.862449 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:20.865984 (+ 3535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:20.870043 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:20.870055 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:20.870073 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:20.870846 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:20.871359 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:20.871367 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:20.871374 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:20.871741 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:20.872292 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:20.872304 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:20.872902 (+ 598us) client_negotiation.cc:769] Sending connection context
0924 10:08:20.873280 (+ 378us) client_negotiation.cc:240] Negotiation successful
0924 10:08:20.873996 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":409,"threads_started":1}
I20260924 10:08:20.896221 14117 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:39293
I20260924 10:08:20.896238 14251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:39293 every 8 connection(s)
I20260924 10:08:20.897616 14117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:08:20.898296 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.883776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:55621 (local address 127.6.2.254:46055)
0924 10:08:20.883994 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:20.884002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:20.884654 (+ 652us) server_negotiation.cc:407] Connection header received
0924 10:08:20.884939 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:20.884948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:20.885069 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:20.885185 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:20.887151 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.888404 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.890534 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.890943 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.891106 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:20.893001 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:20.893006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:20.893009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:20.893042 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:20.895627 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:20.896356 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:20.896364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:20.896370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:20.896533 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:20.896887 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:20.896893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:20.896896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:20.897159 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:20.897316 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:20.897720 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 10:08:20.897934 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:08:20.898481 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.883796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:46055 (local address 127.6.2.195:55621)
0924 10:08:20.884531 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:20.884558 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:20.884753 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:20.885415 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:20.885420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:20.886285 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:20.886882 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:20.886907 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.888605 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.888621 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:20.890385 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:20.890401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.891178 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.891186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:20.891342 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:20.892068 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:20.892085 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:20.892814 (+ 729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:20.895770 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:20.895775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:20.895779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:20.896234 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:20.896637 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:20.896645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:20.896649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:20.896776 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:20.897272 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:20.897276 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:20.897467 (+ 191us) client_negotiation.cc:769] Sending connection context
0924 10:08:20.897608 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 10:08:20.897933 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":399,"threads_started":1}
I20260924 10:08:20.900074 14252 heartbeater.cc:347] Connected to a master server at 127.6.2.254:46055
I20260924 10:08:20.900313 14252 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:20.901070 14252 heartbeater.cc:511] Master 127.6.2.254:46055 requested a full tablet report, sending...
I20260924 10:08:20.901708 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 14117
I20260924 10:08:20.901877 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:08:20.902323 6155 external_mini_cluster.cc:1428] Setting key 36ad01718815651c4b705168606c164c
I20260924 10:08:20.902769 13766 ts_manager.cc:194] Registered new tserver with Master: 6c8d7c65d75b443c81a6a512ded1d1b1 (127.6.2.195:39293)
I20260924 10:08:20.903790 13766 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:53633
I20260924 10:08:20.905325 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:08:20 dist-test-slave-s3n3 krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:08:20 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:20 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:08:20.956923 13862 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:20.946019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.6.2.254:46055)
0924 10:08:20.946263 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:20.946272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:20.946570 (+ 298us) server_negotiation.cc:407] Connection header received
0924 10:08:20.946789 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:20.946793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:20.946855 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:20.946958 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:20.948766 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.949504 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.950800 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.951277 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.951403 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:20.956706 (+ 5303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38840: BlockingRecv error: recv got EOF from 127.0.0.1:38840 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 10:08:20 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:20.973323 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.960090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38856 (local address 127.6.2.254:46055)
0924 10:08:20.960301 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:20.960310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:20.960556 (+ 246us) server_negotiation.cc:407] Connection header received
0924 10:08:20.960749 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:20.960754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:20.960827 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:20.960925 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:20.962041 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.962748 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.964020 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:20.964444 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:20.964581 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:20.968214 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:20.968219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:20.968222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:20.968254 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:20.970720 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:20.971453 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:20.971458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:20.971462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:20.971580 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:20.971941 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:20.971946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:20.971949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:20.972231 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:20.972551 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:20.972792 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 10:08:20.973013 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:08:20.977613 13766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:08:20.984436 13766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:08:21.006670 14270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.996549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:44605 (local address 127.0.0.1:36054)
0924 10:08:20.997553 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:20.997608 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:20.997822 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:20.999006 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:20.999013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:20.999042 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:20.999463 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:20.999475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.001384 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.001389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.003849 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.003863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.005677 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.005683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.005863 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.005921 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 10:08:21.006031 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 10:08:21.006208 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":3,"thread_start_us":1350,"threads_started":1}
I20260924 10:08:21.007447 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.996781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:41261 (local address 127.0.0.1:39804)
0924 10:08:20.998981 (+ 2200us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:20.999002 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:20.999164 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:20.999935 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:20.999939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:20.999964 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:21.000368 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.000380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.002192 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.002198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.004288 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.004302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.006352 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.006359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.006536 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.006567 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:08:21.006670 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 10:08:21.007127 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2067,"thread_start_us":1939,"threads_started":1}
I20260924 10:08:21.007936 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.996812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.6.2.194:44605)
0924 10:08:20.998435 (+ 1623us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:20.998444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:20.998476 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:08:20.998572 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:20.998579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:20.998814 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:20.998997 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:20.999686 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.001237 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.004055 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.005442 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.007200 (+ 1758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.007389 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.007464 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.007656 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1464,"thread_start_us":1358,"threads_started":1}
I20260924 10:08:21.008487 14273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.997356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.6.2.193:41261)
0924 10:08:20.998710 (+ 1354us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:20.998717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:20.999250 (+ 533us) server_negotiation.cc:407] Connection header received
0924 10:08:20.999355 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:20.999363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:20.999569 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:20.999736 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:21.000543 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.002017 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.004454 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.005871 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.005984 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.006201 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.007731 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.007888 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1211,"thread_start_us":1090,"threads_started":1}
I20260924 10:08:21.011711 14043 tablet_service.cc:1467] Processing CreateTablet for tablet f4a2d87691b7433db4975623459f91e1 (DEFAULT_TABLE table=test-table [id=06f200ae46be44a79de1cd76ef45e61c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:21.012631 13904 tablet_service.cc:1467] Processing CreateTablet for tablet f4a2d87691b7433db4975623459f91e1 (DEFAULT_TABLE table=test-table [id=06f200ae46be44a79de1cd76ef45e61c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:21.013176 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a2d87691b7433db4975623459f91e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:21.013919 13904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a2d87691b7433db4975623459f91e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:21.017269 14270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.999183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:39293 (local address 127.0.0.1:52100)
0924 10:08:21.006905 (+ 7722us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:21.006921 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:21.007007 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:21.011419 (+ 4412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:21.011424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:21.011451 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:21.011643 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.011654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.013183 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.013190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.015036 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.015048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.016397 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.016402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.016611 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.016648 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 10:08:21.016778 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 10:08:21.016933 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7642,"thread_start_us":9895,"threads_started":1}
I20260924 10:08:21.017314 14275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:20.999805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.6.2.195:39293)
0924 10:08:21.010872 (+ 11067us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:21.010878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:21.010905 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:08:21.010980 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:21.010985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:21.011238 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:21.011432 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:21.011829 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.013040 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.015250 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.016233 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.016386 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.016642 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.016815 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.016980 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10914,"thread_start_us":10724,"threads_started":1}
I20260924 10:08:21.021169 14182 tablet_service.cc:1467] Processing CreateTablet for tablet f4a2d87691b7433db4975623459f91e1 (DEFAULT_TABLE table=test-table [id=06f200ae46be44a79de1cd76ef45e61c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:21.022451 14182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a2d87691b7433db4975623459f91e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:21.028491 14276 tablet_bootstrap.cc:492] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044: Bootstrap starting.
I20260924 10:08:21.029902 14277 tablet_bootstrap.cc:492] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5: Bootstrap starting.
I20260924 10:08:21.032150 14276 tablet_bootstrap.cc:654] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:21.033113 14277 tablet_bootstrap.cc:654] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:21.033138 14276 log.cc:824] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:21.034299 14277 log.cc:824] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:21.035447 14279 tablet_bootstrap.cc:492] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1: Bootstrap starting.
I20260924 10:08:21.036345 14276 tablet_bootstrap.cc:492] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044: No bootstrap required, opened a new log
I20260924 10:08:21.036703 14276 ts_tablet_manager.cc:1397] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 10:08:21.039013 14279 tablet_bootstrap.cc:654] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:21.040195 14279 log.cc:824] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:21.042774 14279 tablet_bootstrap.cc:492] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1: No bootstrap required, opened a new log
I20260924 10:08:21.043027 14279 ts_tablet_manager.cc:1397] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 10:08:21.043236 14277 tablet_bootstrap.cc:492] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5: No bootstrap required, opened a new log
I20260924 10:08:21.043494 14277 ts_tablet_manager.cc:1397] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260924 10:08:21.047642 14276 raft_consensus.cc:348] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.048063 14276 raft_consensus.cc:374] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:21.048157 14276 raft_consensus.cc:729] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4d5fa0bfa64ca0a5254009350e3044, State: Initialized, Role: FOLLOWER
I20260924 10:08:21.048774 14276 consensus_queue.cc:260] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.049940 14113 heartbeater.cc:503] Master 127.6.2.254:46055 was elected leader, sending a full tablet report...
I20260924 10:08:21.050459 14276 ts_tablet_manager.cc:1428] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260924 10:08:21.051858 14279 raft_consensus.cc:348] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.052189 14279 raft_consensus.cc:374] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:21.052301 14279 raft_consensus.cc:729] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8d7c65d75b443c81a6a512ded1d1b1, State: Initialized, Role: FOLLOWER
I20260924 10:08:21.052850 14279 consensus_queue.cc:260] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.053576 14277 raft_consensus.cc:348] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.053884 14252 heartbeater.cc:503] Master 127.6.2.254:46055 was elected leader, sending a full tablet report...
I20260924 10:08:21.053968 14277 raft_consensus.cc:374] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:21.054059 14277 raft_consensus.cc:729] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86812a9280f479bbcce24732391b5c5, State: Initialized, Role: FOLLOWER
I20260924 10:08:21.054488 14279 ts_tablet_manager.cc:1428] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260924 10:08:21.054605 14277 consensus_queue.cc:260] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.056295 14277 ts_tablet_manager.cc:1428] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
W20260924 10:08:21.061992 13975 tablet.cc:2369] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:08:21.126415 14114 tablet.cc:2369] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:08:21.149896 14253 tablet.cc:2369] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:08:21.224840 14283 raft_consensus.cc:482] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:08:21.225212 14283 raft_consensus.cc:504] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.226899 14283 leader_election.cc:288] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194:44605), f86812a9280f479bbcce24732391b5c5 (127.6.2.193:41261)
I20260924 10:08:21.234083 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.227449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:41261 (local address 127.6.2.195:46215)
0924 10:08:21.227830 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:21.227863 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:21.228054 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:21.228471 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:21.228475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:21.228526 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:21.228829 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.228843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.230002 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.230008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.231776 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.231793 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.233275 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.233280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.233455 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.233478 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:08:21.233633 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 10:08:21.233772 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":1}
I20260924 10:08:21.234145 14273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.227482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:46215 (local address 127.6.2.193:41261)
0924 10:08:21.227698 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:21.227704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:21.227967 (+ 263us) server_negotiation.cc:407] Connection header received
0924 10:08:21.228225 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:21.228231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:21.228322 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:21.228420 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:21.228980 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.229851 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.231929 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.233131 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.233266 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.233366 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.233688 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.233852 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:08:21.234810 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.227492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:59433 (local address 127.6.2.194:44605)
0924 10:08:21.227729 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:21.227736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:21.228427 (+ 691us) server_negotiation.cc:407] Connection header received
0924 10:08:21.228585 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:21.228590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:21.228715 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:21.228836 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:21.229393 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.230318 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.232480 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.233694 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.233833 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.233934 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.234499 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.234606 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:08:21.235057 14285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.227538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:44605 (local address 127.6.2.195:59433)
0924 10:08:21.228289 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:21.228307 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:21.228447 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:21.228848 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:21.228851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:21.228869 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:21.229240 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.229255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.230498 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.230516 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.232219 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.232234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.234098 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.234105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.234298 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.234337 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 10:08:21.234474 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 10:08:21.234634 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":36,"thread_start_us":439,"threads_started":1}
I20260924 10:08:21.235352 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a2d87691b7433db4975623459f91e1" candidate_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86812a9280f479bbcce24732391b5c5" is_pre_election: true
I20260924 10:08:21.236030 13924 raft_consensus.cc:2454] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c8d7c65d75b443c81a6a512ded1d1b1 in term 0.
I20260924 10:08:21.236140 14068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a2d87691b7433db4975623459f91e1" candidate_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" is_pre_election: true
I20260924 10:08:21.236726 14068 raft_consensus.cc:2454] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c8d7c65d75b443c81a6a512ded1d1b1 in term 0.
I20260924 10:08:21.236835 14134 leader_election.cc:302] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c8d7c65d75b443c81a6a512ded1d1b1, f86812a9280f479bbcce24732391b5c5; no voters:
I20260924 10:08:21.237344 14283 raft_consensus.cc:2781] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:08:21.237505 14283 raft_consensus.cc:482] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:08:21.237651 14283 raft_consensus.cc:3037] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:21.240061 14283 raft_consensus.cc:504] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.240671 14283 leader_election.cc:288] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [CANDIDATE]: Term 1 election: Requested vote from peers ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194:44605), f86812a9280f479bbcce24732391b5c5 (127.6.2.193:41261)
I20260924 10:08:21.241120 14068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a2d87691b7433db4975623459f91e1" candidate_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4d5fa0bfa64ca0a5254009350e3044"
I20260924 10:08:21.241243 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a2d87691b7433db4975623459f91e1" candidate_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86812a9280f479bbcce24732391b5c5"
I20260924 10:08:21.241349 14068 raft_consensus.cc:3037] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:21.241438 13924 raft_consensus.cc:3037] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:21.244642 13924 raft_consensus.cc:2454] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c8d7c65d75b443c81a6a512ded1d1b1 in term 1.
I20260924 10:08:21.244642 14068 raft_consensus.cc:2454] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c8d7c65d75b443c81a6a512ded1d1b1 in term 1.
I20260924 10:08:21.245046 14134 leader_election.cc:302] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c8d7c65d75b443c81a6a512ded1d1b1, f86812a9280f479bbcce24732391b5c5; no voters:
I20260924 10:08:21.246232 14283 raft_consensus.cc:2781] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:21.246644 14283 raft_consensus.cc:686] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 LEADER]: Becoming Leader. State: Replica: 6c8d7c65d75b443c81a6a512ded1d1b1, State: Running, Role: LEADER
I20260924 10:08:21.247186 14283 consensus_queue.cc:237] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.252249 13767 catalog_manager.cc:5495] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c8d7c65d75b443c81a6a512ded1d1b1 (127.6.2.195). New cstate: current_term: 1 leader_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } health_report { overall_health: HEALTHY } } }
I20260924 10:08:21.302235 14275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.297461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52104 (local address 127.6.2.195:39293)
0924 10:08:21.297630 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:21.297635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:21.297652 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:08:21.297767 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:21.297771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:21.297844 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:21.297965 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:08:21.298247 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.298947 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.300310 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.300718 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.300813 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.301154 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:08:21.301699 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:08:21.301804 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.301965 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.302031 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:08:21.310719 13924 raft_consensus.cc:1261] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Refusing update from remote peer 6c8d7c65d75b443c81a6a512ded1d1b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:21.310719 14068 raft_consensus.cc:1261] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Refusing update from remote peer 6c8d7c65d75b443c81a6a512ded1d1b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:21.311777 14286 consensus_queue.cc:1035] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:21.312157 14283 consensus_queue.cc:1035] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:21.335656 13765 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:21.335913 13765 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:21.339788 13765 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b: Sending DeleteTablet for 3 replicas of tablet f4a2d87691b7433db4975623459f91e1
I20260924 10:08:21.340865 14043 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a2d87691b7433db4975623459f91e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:21 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:36054
I20260924 10:08:21.340868 13904 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a2d87691b7433db4975623459f91e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:21 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:39804
I20260924 10:08:21.341085 14182 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a2d87691b7433db4975623459f91e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:21 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:52100
I20260924 10:08:21.343387 13765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:08:21.345419 13765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:08:21.353821 14300 tablet_replica.cc:330] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5: stopping tablet replica
I20260924 10:08:21.353713 14042 tablet_service.cc:1467] Processing CreateTablet for tablet 57b17cde51bb4d2b88d7f0b09ccff526 (DEFAULT_TABLE table=test-table [id=6c7a10a18f2d476aa07b940608fd6771]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:21.353734 14181 tablet_service.cc:1467] Processing CreateTablet for tablet 57b17cde51bb4d2b88d7f0b09ccff526 (DEFAULT_TABLE table=test-table [id=6c7a10a18f2d476aa07b940608fd6771]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:21.353717 13903 tablet_service.cc:1467] Processing CreateTablet for tablet 57b17cde51bb4d2b88d7f0b09ccff526 (DEFAULT_TABLE table=test-table [id=6c7a10a18f2d476aa07b940608fd6771]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:21.354250 14181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b17cde51bb4d2b88d7f0b09ccff526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:21.354285 14042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b17cde51bb4d2b88d7f0b09ccff526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:21.354283 13903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b17cde51bb4d2b88d7f0b09ccff526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:21.354321 14300 raft_consensus.cc:2229] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:21.354821 14300 raft_consensus.cc:2258] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:21.355953 14299 tablet_replica.cc:330] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044: stopping tablet replica
I20260924 10:08:21.356352 14299 raft_consensus.cc:2229] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:21.356823 14299 raft_consensus.cc:2258] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:21.359431 14300 ts_tablet_manager.cc:1905] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:21.360946 14279 tablet_bootstrap.cc:492] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1: Bootstrap starting.
I20260924 10:08:21.362114 14276 tablet_bootstrap.cc:492] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044: Bootstrap starting.
I20260924 10:08:21.362947 14279 tablet_bootstrap.cc:654] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:21.363770 14276 tablet_bootstrap.cc:654] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:21.366598 14300 ts_tablet_manager.cc:1918] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:21.366771 14300 log.cc:1196] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/f4a2d87691b7433db4975623459f91e1
I20260924 10:08:21.367271 14277 tablet_bootstrap.cc:492] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5: Bootstrap starting.
I20260924 10:08:21.367451 14300 ts_tablet_manager.cc:1939] T f4a2d87691b7433db4975623459f91e1 P f86812a9280f479bbcce24732391b5c5: Deleting consensus metadata
I20260924 10:08:21.369329 14277 tablet_bootstrap.cc:654] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:21.376983 14276 tablet_bootstrap.cc:492] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044: No bootstrap required, opened a new log
I20260924 10:08:21.377138 14276 ts_tablet_manager.cc:1397] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20260924 10:08:21.378458 14299 ts_tablet_manager.cc:1905] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:21.379863 14301 tablet_replica.cc:330] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1: stopping tablet replica
I20260924 10:08:21.380272 14301 raft_consensus.cc:2229] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:08:21.380880 14301 raft_consensus.cc:2258] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:21.382704 14301 ts_tablet_manager.cc:1905] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:21.379040 14276 raft_consensus.cc:348] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.387174 14276 raft_consensus.cc:374] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:21.387285 14276 raft_consensus.cc:729] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4d5fa0bfa64ca0a5254009350e3044, State: Initialized, Role: FOLLOWER
I20260924 10:08:21.387534 14276 consensus_queue.cc:260] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.388136 14276 ts_tablet_manager.cc:1428] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20260924 10:08:21.390033 14301 ts_tablet_manager.cc:1918] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:21.390200 14301 log.cc:1196] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/f4a2d87691b7433db4975623459f91e1
I20260924 10:08:21.390751 14301 ts_tablet_manager.cc:1939] T f4a2d87691b7433db4975623459f91e1 P 6c8d7c65d75b443c81a6a512ded1d1b1: Deleting consensus metadata
I20260924 10:08:21.390792 14277 tablet_bootstrap.cc:492] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5: No bootstrap required, opened a new log
I20260924 10:08:21.391038 14277 ts_tablet_manager.cc:1397] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.006s
I20260924 10:08:21.391981 14277 raft_consensus.cc:348] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.392294 14277 raft_consensus.cc:374] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:21.392447 14277 raft_consensus.cc:729] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86812a9280f479bbcce24732391b5c5, State: Initialized, Role: FOLLOWER
I20260924 10:08:21.392459 13752 catalog_manager.cc:4841] TS 6c8d7c65d75b443c81a6a512ded1d1b1 (127.6.2.195:39293): tablet f4a2d87691b7433db4975623459f91e1 (table test-table [id=06f200ae46be44a79de1cd76ef45e61c]) successfully deleted
I20260924 10:08:21.392904 14277 consensus_queue.cc:260] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.394086 14277 ts_tablet_manager.cc:1428] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 10:08:21.395759 14279 tablet_bootstrap.cc:492] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1: No bootstrap required, opened a new log
I20260924 10:08:21.395958 14279 ts_tablet_manager.cc:1397] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.005s
I20260924 10:08:21.396342 14299 ts_tablet_manager.cc:1918] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:21.396452 13752 catalog_manager.cc:4841] TS f86812a9280f479bbcce24732391b5c5 (127.6.2.193:41261): tablet f4a2d87691b7433db4975623459f91e1 (table test-table [id=06f200ae46be44a79de1cd76ef45e61c]) successfully deleted
I20260924 10:08:21.396590 14299 log.cc:1196] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/f4a2d87691b7433db4975623459f91e1
I20260924 10:08:21.397063 14299 ts_tablet_manager.cc:1939] T f4a2d87691b7433db4975623459f91e1 P ca4d5fa0bfa64ca0a5254009350e3044: Deleting consensus metadata
I20260924 10:08:21.396973 14279 raft_consensus.cc:348] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.397188 14279 raft_consensus.cc:374] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:21.397300 14279 raft_consensus.cc:729] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8d7c65d75b443c81a6a512ded1d1b1, State: Initialized, Role: FOLLOWER
I20260924 10:08:21.397595 14279 consensus_queue.cc:260] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.397867 14282 raft_consensus.cc:482] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:08:21.398162 14279 ts_tablet_manager.cc:1428] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 10:08:21.398046 14282 raft_consensus.cc:504] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.399850 14282 leader_election.cc:288] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f86812a9280f479bbcce24732391b5c5 (127.6.2.193:41261), 6c8d7c65d75b443c81a6a512ded1d1b1 (127.6.2.195:39293)
I20260924 10:08:21.401474 13754 catalog_manager.cc:4841] TS ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194:44605): tablet f4a2d87691b7433db4975623459f91e1 (table test-table [id=06f200ae46be44a79de1cd76ef45e61c]) successfully deleted
I20260924 10:08:21.410976 14273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.400425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:47529 (local address 127.6.2.193:41261)
0924 10:08:21.400623 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:21.400630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:21.404636 (+ 4006us) server_negotiation.cc:407] Connection header received
0924 10:08:21.404870 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:21.404876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:21.404944 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:21.404998 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:21.405821 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.406668 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.409060 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.409842 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.409996 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.410104 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.410573 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.410704 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:08:21.411823 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.400039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:41261 (local address 127.6.2.194:47529)
0924 10:08:21.404540 (+ 4501us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:21.404559 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:21.404731 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:21.405203 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:21.405207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:21.405236 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:21.405625 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.405636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.406810 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.406813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.408918 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.408928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.410065 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.410071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.410373 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.410411 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 10:08:21.411443 (+ 1032us) client_negotiation.cc:240] Negotiation successful
0924 10:08:21.411543 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4365,"mutex_wait_us":3,"thread_start_us":4225,"threads_started":1}
I20260924 10:08:21.412459 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b17cde51bb4d2b88d7f0b09ccff526" candidate_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86812a9280f479bbcce24732391b5c5" is_pre_election: true
I20260924 10:08:21.412704 13924 raft_consensus.cc:2454] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca4d5fa0bfa64ca0a5254009350e3044 in term 0.
I20260924 10:08:21.413156 13995 leader_election.cc:302] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca4d5fa0bfa64ca0a5254009350e3044, f86812a9280f479bbcce24732391b5c5; no voters:
I20260924 10:08:21.413532 14282 raft_consensus.cc:2781] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:08:21.413650 14282 raft_consensus.cc:482] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:08:21.413760 14282 raft_consensus.cc:3037] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:21.414924 14275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.404761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:54791 (local address 127.6.2.195:39293)
0924 10:08:21.404914 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:21.404921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:21.407693 (+ 2772us) server_negotiation.cc:407] Connection header received
0924 10:08:21.407876 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:21.407881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:21.407955 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:21.408050 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:21.408651 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.409804 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.412397 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.413722 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.413820 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.413909 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.414500 (+ 591us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.414645 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:08:21.415174 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.405059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:39293 (local address 127.6.2.194:54791)
0924 10:08:21.407573 (+ 2514us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:21.407591 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:21.407729 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:21.408100 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:21.408105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:21.408130 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:21.408496 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.408508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.409971 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.409976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.412203 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.412219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.414113 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.414121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.414317 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.414356 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 10:08:21.414458 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 10:08:21.414599 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2371,"mutex_wait_us":21,"thread_start_us":2169,"threads_started":1}
I20260924 10:08:21.416007 14202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b17cde51bb4d2b88d7f0b09ccff526" candidate_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" is_pre_election: true
I20260924 10:08:21.416335 14202 raft_consensus.cc:2454] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca4d5fa0bfa64ca0a5254009350e3044 in term 0.
I20260924 10:08:21.416797 14282 raft_consensus.cc:504] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.417414 14282 leader_election.cc:288] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [CANDIDATE]: Term 1 election: Requested vote from peers f86812a9280f479bbcce24732391b5c5 (127.6.2.193:41261), 6c8d7c65d75b443c81a6a512ded1d1b1 (127.6.2.195:39293)
I20260924 10:08:21.417768 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b17cde51bb4d2b88d7f0b09ccff526" candidate_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86812a9280f479bbcce24732391b5c5"
I20260924 10:08:21.417975 13924 raft_consensus.cc:3037] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:21.418089 14202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b17cde51bb4d2b88d7f0b09ccff526" candidate_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1"
I20260924 10:08:21.418291 14202 raft_consensus.cc:3037] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:21.420061 13924 raft_consensus.cc:2454] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca4d5fa0bfa64ca0a5254009350e3044 in term 1.
I20260924 10:08:21.420406 13995 leader_election.cc:302] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca4d5fa0bfa64ca0a5254009350e3044, f86812a9280f479bbcce24732391b5c5; no voters:
I20260924 10:08:21.420708 14282 raft_consensus.cc:2781] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:21.421463 14202 raft_consensus.cc:2454] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca4d5fa0bfa64ca0a5254009350e3044 in term 1.
I20260924 10:08:21.422274 14282 raft_consensus.cc:686] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 LEADER]: Becoming Leader. State: Replica: ca4d5fa0bfa64ca0a5254009350e3044, State: Running, Role: LEADER
I20260924 10:08:21.422654 14282 consensus_queue.cc:237] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.425920 13765 catalog_manager.cc:5495] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194). New cstate: current_term: 1 leader_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } health_report { overall_health: UNKNOWN } } }
Sep 24 10:08:21 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244499, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.254@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:21.489464 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.465549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:46055 (local address 127.0.0.1:38868)
0924 10:08:21.466226 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:21.466255 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:21.466406 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:21.466790 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:21.466795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:21.467718 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:21.468040 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.468053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.469027 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.469033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.470559 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.470573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.471367 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.471375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.471569 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.480687 (+ 9118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:21.480715 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:21.483445 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:21.486621 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:21.486636 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:21.486645 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:21.487169 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:21.487554 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:21.487561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:21.487565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:21.487709 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:21.488252 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:21.488261 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:21.488444 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 10:08:21.488621 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 10:08:21.488946 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":381,"threads_started":1}
I20260924 10:08:21.489491 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.465557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.6.2.254:46055)
0924 10:08:21.465819 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:21.465828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:21.466339 (+ 511us) server_negotiation.cc:407] Connection header received
0924 10:08:21.466536 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:21.466539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:21.466614 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:21.466734 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:21.468186 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.468901 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.470731 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.471190 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.471283 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.483678 (+ 12395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:21.483684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:21.483687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:21.483714 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:21.486362 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:21.487287 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:21.487294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:21.487301 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:21.487440 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:21.487823 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:21.487828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:21.487831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:21.488122 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:21.488281 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.488728 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.489063 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":112}
I20260924 10:08:21.494809 13765 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:38868:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:08:21.495926 13765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:08:21.502772 13903 tablet_service.cc:1467] Processing CreateTablet for tablet 6097b23babf544e6815c6e807613b21d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0623d938e8204a4e8340141d26f16802]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:08:21.502806 14043 tablet_service.cc:1467] Processing CreateTablet for tablet 6097b23babf544e6815c6e807613b21d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0623d938e8204a4e8340141d26f16802]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:08:21.503321 13903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6097b23babf544e6815c6e807613b21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:21.503343 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6097b23babf544e6815c6e807613b21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:21.503152 14182 tablet_service.cc:1467] Processing CreateTablet for tablet 6097b23babf544e6815c6e807613b21d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0623d938e8204a4e8340141d26f16802]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:08:21.503696 14182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6097b23babf544e6815c6e807613b21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:21.510790 14276 tablet_bootstrap.cc:492] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044: Bootstrap starting.
I20260924 10:08:21.510836 14277 tablet_bootstrap.cc:492] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5: Bootstrap starting.
I20260924 10:08:21.512135 14276 tablet_bootstrap.cc:654] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:21.512513 14277 tablet_bootstrap.cc:654] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:21.514104 14276 tablet_bootstrap.cc:492] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044: No bootstrap required, opened a new log
I20260924 10:08:21.514278 14276 ts_tablet_manager.cc:1397] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 10:08:21.514855 14276 raft_consensus.cc:348] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.515058 14276 raft_consensus.cc:374] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:21.515154 14276 raft_consensus.cc:729] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4d5fa0bfa64ca0a5254009350e3044, State: Initialized, Role: FOLLOWER
I20260924 10:08:21.515389 14276 consensus_queue.cc:260] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.515775 14282 tablet_replica.cc:439] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } } }
I20260924 10:08:21.516031 14282 tablet_replica.cc:442] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:08:21.516265 14279 tablet_bootstrap.cc:492] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1: Bootstrap starting.
I20260924 10:08:21.516784 14276 ts_tablet_manager.cc:1428] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:08:21.517985 14279 tablet_bootstrap.cc:654] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:21.520318 14279 tablet_bootstrap.cc:492] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1: No bootstrap required, opened a new log
I20260924 10:08:21.520560 14279 ts_tablet_manager.cc:1397] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260924 10:08:21.520583 14277 tablet_bootstrap.cc:492] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5: No bootstrap required, opened a new log
I20260924 10:08:21.520748 14277 ts_tablet_manager.cc:1397] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260924 10:08:21.521428 14279 raft_consensus.cc:348] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.521646 14279 raft_consensus.cc:374] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:21.521513 14277 raft_consensus.cc:348] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.521740 14279 raft_consensus.cc:729] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8d7c65d75b443c81a6a512ded1d1b1, State: Initialized, Role: FOLLOWER
I20260924 10:08:21.521750 14277 raft_consensus.cc:374] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:21.521817 14277 raft_consensus.cc:729] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86812a9280f479bbcce24732391b5c5, State: Initialized, Role: FOLLOWER
I20260924 10:08:21.521977 14279 consensus_queue.cc:260] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.522064 14277 consensus_queue.cc:260] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.522370 14283 tablet_replica.cc:439] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } } }
I20260924 10:08:21.522624 14283 tablet_replica.cc:442] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:08:21.522548 14284 tablet_replica.cc:439] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } } }
I20260924 10:08:21.522872 14284 tablet_replica.cc:442] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:08:21.523130 14279 ts_tablet_manager.cc:1428] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:08:21.523795 14277 ts_tablet_manager.cc:1428] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 10:08:21.613554 14283 raft_consensus.cc:482] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:08:21.613787 14283 raft_consensus.cc:504] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.614502 14283 leader_election.cc:288] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f86812a9280f479bbcce24732391b5c5 (127.6.2.193:41261), ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194:44605)
I20260924 10:08:21.615024 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6097b23babf544e6815c6e807613b21d" candidate_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86812a9280f479bbcce24732391b5c5" is_pre_election: true
I20260924 10:08:21.615166 14068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6097b23babf544e6815c6e807613b21d" candidate_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" is_pre_election: true
I20260924 10:08:21.615322 13924 raft_consensus.cc:2454] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c8d7c65d75b443c81a6a512ded1d1b1 in term 0.
I20260924 10:08:21.615393 14068 raft_consensus.cc:2454] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c8d7c65d75b443c81a6a512ded1d1b1 in term 0.
I20260924 10:08:21.615844 14134 leader_election.cc:302] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c8d7c65d75b443c81a6a512ded1d1b1, f86812a9280f479bbcce24732391b5c5; no voters:
I20260924 10:08:21.616128 14283 raft_consensus.cc:2781] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:08:21.616258 14283 raft_consensus.cc:482] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:08:21.616355 14283 raft_consensus.cc:3037] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:21.619663 14283 raft_consensus.cc:504] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.620340 14283 leader_election.cc:288] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [CANDIDATE]: Term 1 election: Requested vote from peers f86812a9280f479bbcce24732391b5c5 (127.6.2.193:41261), ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194:44605)
I20260924 10:08:21.620850 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6097b23babf544e6815c6e807613b21d" candidate_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86812a9280f479bbcce24732391b5c5"
I20260924 10:08:21.620860 14068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6097b23babf544e6815c6e807613b21d" candidate_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4d5fa0bfa64ca0a5254009350e3044"
I20260924 10:08:21.621069 14068 raft_consensus.cc:3037] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:21.621116 13924 raft_consensus.cc:3037] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:21.623600 13924 raft_consensus.cc:2454] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c8d7c65d75b443c81a6a512ded1d1b1 in term 1.
I20260924 10:08:21.624025 14134 leader_election.cc:302] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c8d7c65d75b443c81a6a512ded1d1b1, f86812a9280f479bbcce24732391b5c5; no voters:
I20260924 10:08:21.624079 14068 raft_consensus.cc:2454] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c8d7c65d75b443c81a6a512ded1d1b1 in term 1.
I20260924 10:08:21.624353 14283 raft_consensus.cc:2781] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:21.624516 14283 raft_consensus.cc:686] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 LEADER]: Becoming Leader. State: Replica: 6c8d7c65d75b443c81a6a512ded1d1b1, State: Running, Role: LEADER
I20260924 10:08:21.624825 14283 consensus_queue.cc:237] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.626655 14294 tablet_replica.cc:439] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1: TxnStatusTablet state changed. Reason: New leader 6c8d7c65d75b443c81a6a512ded1d1b1. Latest consensus state: current_term: 1 leader_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } } }
I20260924 10:08:21.627077 14294 tablet_replica.cc:442] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:08:21.628152 14319 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:08:21.628415 14319 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:08:21.628409 13765 catalog_manager.cc:5495] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c8d7c65d75b443c81a6a512ded1d1b1 (127.6.2.195). New cstate: current_term: 1 leader_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } health_report { overall_health: HEALTHY } } }
I20260924 10:08:21.653677 14275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.648042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.6.2.195:39293)
0924 10:08:21.648217 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:21.648222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:21.648239 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:08:21.648504 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:21.648509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:21.648593 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:21.648671 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:08:21.649106 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.649910 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.651862 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.652461 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.652565 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.653065 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:08:21.653181 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:08:21.653250 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.653407 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.653464 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 10:08:21.653839 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.647861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:39293 (local address 127.0.0.1:52118)
0924 10:08:21.648158 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:21.648180 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:21.648328 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:21.648724 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:21.648729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:21.648742 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:08:21.648960 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.648973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.650056 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.650062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.651687 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.651704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.652631 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.652637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.652816 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.652888 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:08:21.653283 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:08:21.653301 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 10:08:21.653391 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 10:08:21.653544 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 10:08:21.657476 13924 raft_consensus.cc:1261] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Refusing update from remote peer 6c8d7c65d75b443c81a6a512ded1d1b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:21.657460 14068 raft_consensus.cc:1261] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Refusing update from remote peer 6c8d7c65d75b443c81a6a512ded1d1b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:21.658041 14283 consensus_queue.cc:1035] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:21.658385 14294 consensus_queue.cc:1035] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:21.663895 14282 tablet_replica.cc:439] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044: TxnStatusTablet state changed. Reason: New leader 6c8d7c65d75b443c81a6a512ded1d1b1. Latest consensus state: current_term: 1 leader_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } } }
I20260924 10:08:21.664207 14282 tablet_replica.cc:442] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:08:21.665297 14284 tablet_replica.cc:439] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5: TxnStatusTablet state changed. Reason: New leader 6c8d7c65d75b443c81a6a512ded1d1b1. Latest consensus state: current_term: 1 leader_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } } }
I20260924 10:08:21.665568 14284 tablet_replica.cc:442] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:08:21.666680 14294 tablet_replica.cc:439] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } } }
I20260924 10:08:21.666946 14294 tablet_replica.cc:442] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:08:21.667423 14283 tablet_replica.cc:439] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } } }
I20260924 10:08:21.667460 14289 mvcc.cc:205] Tried to move back new op lower bound from 7332841478782656512 to 7332841478657695744. Current Snapshot: MvccSnapshot[applied={T|T < 7332841478782656512 or (T in {7332841478782656512})}]
I20260924 10:08:21.667706 14283 tablet_replica.cc:442] T 6097b23babf544e6815c6e807613b21d P 6c8d7c65d75b443c81a6a512ded1d1b1: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:08:21.669523 14282 tablet_replica.cc:439] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } } }
I20260924 10:08:21.669798 14282 tablet_replica.cc:442] T 6097b23babf544e6815c6e807613b21d P ca4d5fa0bfa64ca0a5254009350e3044: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:08:21.671244 14284 tablet_replica.cc:439] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } } }
I20260924 10:08:21.671480 14284 tablet_replica.cc:442] T 6097b23babf544e6815c6e807613b21d P f86812a9280f479bbcce24732391b5c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:08:21.681092 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.675005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.6.2.194:44605)
0924 10:08:21.675207 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:21.675213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:21.675346 (+ 133us) server_negotiation.cc:407] Connection header received
0924 10:08:21.675565 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:21.675568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:21.675649 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:21.675765 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:08:21.676251 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.676972 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.678875 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.679311 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.679478 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.679930 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:08:21.680482 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:08:21.680584 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.680734 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.680802 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 10:08:21.697494 14275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.691483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:51315 (local address 127.6.2.195:39293)
0924 10:08:21.691711 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:21.691717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:21.692042 (+ 325us) server_negotiation.cc:407] Connection header received
0924 10:08:21.692224 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:21.692227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:21.692289 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:21.692370 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:08:21.692915 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.693696 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.695722 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.696197 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.696289 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.696801 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:08:21.696922 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:08:21.696994 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.697165 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.697228 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 10:08:21.697696 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.691309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:39293 (local address 127.6.2.194:51315)
0924 10:08:21.691929 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:21.691950 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:21.692106 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:21.692410 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:21.692416 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:21.692428 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:08:21.692747 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.692760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.693834 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.693839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.695583 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.695598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.696377 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.696384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.696557 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.696638 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:08:21.697029 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:08:21.697041 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 10:08:21.697134 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 10:08:21.697291 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":284,"threads_started":1}
I20260924 10:08:21.707298 13924 raft_consensus.cc:1261] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Refusing update from remote peer ca4d5fa0bfa64ca0a5254009350e3044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:21.707553 14202 raft_consensus.cc:1261] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Refusing update from remote peer ca4d5fa0bfa64ca0a5254009350e3044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:21.707875 14282 consensus_queue.cc:1035] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:21.708462 14282 consensus_queue.cc:1035] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:21.715868 14291 mvcc.cc:205] Tried to move back new op lower bound from 7332841478986911744 to 7332841477830922240. Current Snapshot: MvccSnapshot[applied={T|T < 7332841478986911744}]
I20260924 10:08:21.747205 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.740697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:59509 (local address 127.6.2.194:44605)
0924 10:08:21.741071 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:21.741078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:21.741479 (+ 401us) server_negotiation.cc:407] Connection header received
0924 10:08:21.741754 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:21.741761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:21.741845 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:21.741935 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:08:21.742688 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.743548 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.745189 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.745790 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.745909 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.746402 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:08:21.746527 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:08:21.746590 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.746818 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.746892 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260924 10:08:21.747334 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.740681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:44605 (local address 127.6.2.195:59509)
0924 10:08:21.741376 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:21.741397 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:21.741604 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:21.742084 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:21.742087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:21.742096 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:08:21.742506 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.742520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.743676 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.743684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.745046 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:21.745058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.745941 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.745948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:21.746106 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.746190 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:08:21.746666 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:08:21.746679 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 10:08:21.746793 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 10:08:21.746922 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":590,"threads_started":1}
I20260924 10:08:21.782984 13766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:21.783432 13766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:21.787281 13766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b: Sending DeleteTablet for 3 replicas of tablet 57b17cde51bb4d2b88d7f0b09ccff526
I20260924 10:08:21.788250 14043 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b17cde51bb4d2b88d7f0b09ccff526 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:21 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:36054
I20260924 10:08:21.788556 14299 tablet_replica.cc:330] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044: stopping tablet replica
I20260924 10:08:21.788794 14299 raft_consensus.cc:2229] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:08:21.788868 13903 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b17cde51bb4d2b88d7f0b09ccff526 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:21 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:39804
I20260924 10:08:21.789197 14300 tablet_replica.cc:330] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5: stopping tablet replica
I20260924 10:08:21.789352 14299 raft_consensus.cc:2258] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:21.789420 14300 raft_consensus.cc:2229] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:21.789635 14300 raft_consensus.cc:2258] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:21.790755 14299 ts_tablet_manager.cc:1905] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:21.791051 14300 ts_tablet_manager.cc:1905] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:21.795460 14182 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b17cde51bb4d2b88d7f0b09ccff526 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:21 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:52100
I20260924 10:08:21.795775 14301 tablet_replica.cc:330] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1: stopping tablet replica
I20260924 10:08:21.795985 14301 raft_consensus.cc:2229] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:21.796221 14301 raft_consensus.cc:2258] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:21.797609 14301 ts_tablet_manager.cc:1905] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:21.798700 14299 ts_tablet_manager.cc:1918] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:08:21.798904 14299 log.cc:1196] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/57b17cde51bb4d2b88d7f0b09ccff526
I20260924 10:08:21.799464 14299 ts_tablet_manager.cc:1939] T 57b17cde51bb4d2b88d7f0b09ccff526 P ca4d5fa0bfa64ca0a5254009350e3044: Deleting consensus metadata
I20260924 10:08:21.800834 13754 catalog_manager.cc:4841] TS ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194:44605): tablet 57b17cde51bb4d2b88d7f0b09ccff526 (table test-table [id=6c7a10a18f2d476aa07b940608fd6771]) successfully deleted
I20260924 10:08:21.814359 14301 ts_tablet_manager.cc:1918] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:08:21.818747 14300 ts_tablet_manager.cc:1918] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:08:21.819044 14300 log.cc:1196] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/57b17cde51bb4d2b88d7f0b09ccff526
I20260924 10:08:21.819639 14300 ts_tablet_manager.cc:1939] T 57b17cde51bb4d2b88d7f0b09ccff526 P f86812a9280f479bbcce24732391b5c5: Deleting consensus metadata
I20260924 10:08:21.819872 14301 log.cc:1196] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/57b17cde51bb4d2b88d7f0b09ccff526
I20260924 10:08:21.820590 14301 ts_tablet_manager.cc:1939] T 57b17cde51bb4d2b88d7f0b09ccff526 P 6c8d7c65d75b443c81a6a512ded1d1b1: Deleting consensus metadata
I20260924 10:08:21.822913 13752 catalog_manager.cc:4841] TS f86812a9280f479bbcce24732391b5c5 (127.6.2.193:41261): tablet 57b17cde51bb4d2b88d7f0b09ccff526 (table test-table [id=6c7a10a18f2d476aa07b940608fd6771]) successfully deleted
I20260924 10:08:21.823597 13752 catalog_manager.cc:4841] TS 6c8d7c65d75b443c81a6a512ded1d1b1 (127.6.2.195:39293): tablet 57b17cde51bb4d2b88d7f0b09ccff526 (table test-table [id=6c7a10a18f2d476aa07b940608fd6771]) successfully deleted
Sep 24 10:08:21 dist-test-slave-s3n3 krb5kdc[13722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:08:21 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:21 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:08:21.874131 13862 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:21.863705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.6.2.254:46055)
0924 10:08:21.863953 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:21.863963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:21.864505 (+ 542us) server_negotiation.cc:407] Connection header received
0924 10:08:21.864850 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:21.864856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:21.864934 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:21.865065 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:21.866422 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.867408 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.868723 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.869194 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.869324 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.873933 (+ 4609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38880: BlockingRecv error: recv got EOF from 127.0.0.1:38880 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 10:08:21 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM
I20260924 10:08:21.891948 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:21.877765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38892 (local address 127.6.2.254:46055)
0924 10:08:21.877941 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:21.877946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:21.878353 (+ 407us) server_negotiation.cc:407] Connection header received
0924 10:08:21.878598 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:21.878604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:21.878684 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:21.878784 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:21.880102 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.880810 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.882022 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:21.882436 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:21.882575 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:21.886381 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:21.886387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:21.886390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:21.886427 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:21.889018 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:21.889848 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:21.889852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:21.889855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:21.889982 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:21.890425 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:21.890430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:21.890438 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:21.890712 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:21.890956 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:21.891346 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 10:08:21.891582 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:08:21.896212 13766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:08:21.896926 13766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:08:21.904536 13903 tablet_service.cc:1467] Processing CreateTablet for tablet d435561f445240bdb5866d166426b90a (DEFAULT_TABLE table=test-table [id=707d56b67fdf47d8b22f0656a23df686]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:21.905035 13903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d435561f445240bdb5866d166426b90a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:21.911746 14043 tablet_service.cc:1467] Processing CreateTablet for tablet d435561f445240bdb5866d166426b90a (DEFAULT_TABLE table=test-table [id=707d56b67fdf47d8b22f0656a23df686]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:21.912206 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d435561f445240bdb5866d166426b90a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:21.915966 14182 tablet_service.cc:1467] Processing CreateTablet for tablet d435561f445240bdb5866d166426b90a (DEFAULT_TABLE table=test-table [id=707d56b67fdf47d8b22f0656a23df686]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:21.916420 14182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d435561f445240bdb5866d166426b90a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:21.921478 14277 tablet_bootstrap.cc:492] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5: Bootstrap starting.
I20260924 10:08:21.923219 14277 tablet_bootstrap.cc:654] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:21.928956 14276 tablet_bootstrap.cc:492] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044: Bootstrap starting.
I20260924 10:08:21.930943 14276 tablet_bootstrap.cc:654] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:21.931011 14279 tablet_bootstrap.cc:492] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1: Bootstrap starting.
I20260924 10:08:21.933229 14277 tablet_bootstrap.cc:492] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5: No bootstrap required, opened a new log
I20260924 10:08:21.933264 14279 tablet_bootstrap.cc:654] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:21.933391 14277 ts_tablet_manager.cc:1397] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.002s
I20260924 10:08:21.934181 14277 raft_consensus.cc:348] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.934440 14277 raft_consensus.cc:374] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:21.934531 14277 raft_consensus.cc:729] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86812a9280f479bbcce24732391b5c5, State: Initialized, Role: FOLLOWER
I20260924 10:08:21.934787 14277 consensus_queue.cc:260] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.937899 14277 ts_tablet_manager.cc:1428] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260924 10:08:21.941447 14276 tablet_bootstrap.cc:492] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044: No bootstrap required, opened a new log
I20260924 10:08:21.941612 14276 ts_tablet_manager.cc:1397] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20260924 10:08:21.942480 14276 raft_consensus.cc:348] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.942775 14276 raft_consensus.cc:374] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:21.942875 14276 raft_consensus.cc:729] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4d5fa0bfa64ca0a5254009350e3044, State: Initialized, Role: FOLLOWER
I20260924 10:08:21.943123 14276 consensus_queue.cc:260] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.944298 14276 ts_tablet_manager.cc:1428] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:08:21.945708 14279 tablet_bootstrap.cc:492] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1: No bootstrap required, opened a new log
I20260924 10:08:21.945868 14279 ts_tablet_manager.cc:1397] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.007s
I20260924 10:08:21.946635 14279 raft_consensus.cc:348] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.946903 14279 raft_consensus.cc:374] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:21.947006 14279 raft_consensus.cc:729] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8d7c65d75b443c81a6a512ded1d1b1, State: Initialized, Role: FOLLOWER
I20260924 10:08:21.947331 14279 consensus_queue.cc:260] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:21.949939 14279 ts_tablet_manager.cc:1428] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260924 10:08:22.131242 14286 raft_consensus.cc:482] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:08:22.131501 14286 raft_consensus.cc:504] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:22.132333 14286 leader_election.cc:288] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f86812a9280f479bbcce24732391b5c5 (127.6.2.193:41261), ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194:44605)
I20260924 10:08:22.132995 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d435561f445240bdb5866d166426b90a" candidate_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86812a9280f479bbcce24732391b5c5" is_pre_election: true
I20260924 10:08:22.133147 14068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d435561f445240bdb5866d166426b90a" candidate_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" is_pre_election: true
I20260924 10:08:22.133263 13924 raft_consensus.cc:2454] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c8d7c65d75b443c81a6a512ded1d1b1 in term 0.
I20260924 10:08:22.133428 14068 raft_consensus.cc:2454] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c8d7c65d75b443c81a6a512ded1d1b1 in term 0.
I20260924 10:08:22.133960 14134 leader_election.cc:302] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c8d7c65d75b443c81a6a512ded1d1b1, f86812a9280f479bbcce24732391b5c5; no voters:
I20260924 10:08:22.134344 14286 raft_consensus.cc:2781] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:08:22.134495 14286 raft_consensus.cc:482] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:08:22.134617 14286 raft_consensus.cc:3037] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:22.138059 14286 raft_consensus.cc:504] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:22.139199 14286 leader_election.cc:288] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [CANDIDATE]: Term 1 election: Requested vote from peers f86812a9280f479bbcce24732391b5c5 (127.6.2.193:41261), ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194:44605)
I20260924 10:08:22.139497 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d435561f445240bdb5866d166426b90a" candidate_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f86812a9280f479bbcce24732391b5c5"
I20260924 10:08:22.139740 13924 raft_consensus.cc:3037] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:22.140048 14068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d435561f445240bdb5866d166426b90a" candidate_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4d5fa0bfa64ca0a5254009350e3044"
I20260924 10:08:22.140273 14068 raft_consensus.cc:3037] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:22.142863 13924 raft_consensus.cc:2454] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c8d7c65d75b443c81a6a512ded1d1b1 in term 1.
I20260924 10:08:22.143455 14068 raft_consensus.cc:2454] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c8d7c65d75b443c81a6a512ded1d1b1 in term 1.
I20260924 10:08:22.143469 14134 leader_election.cc:302] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c8d7c65d75b443c81a6a512ded1d1b1, f86812a9280f479bbcce24732391b5c5; no voters:
I20260924 10:08:22.143857 14286 raft_consensus.cc:2781] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:22.144027 14286 raft_consensus.cc:686] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 LEADER]: Becoming Leader. State: Replica: 6c8d7c65d75b443c81a6a512ded1d1b1, State: Running, Role: LEADER
I20260924 10:08:22.144295 14286 consensus_queue.cc:237] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:22.149053 13765 catalog_manager.cc:5495] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c8d7c65d75b443c81a6a512ded1d1b1 (127.6.2.195). New cstate: current_term: 1 leader_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } health_report { overall_health: HEALTHY } } }
I20260924 10:08:22.231523 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:22.223116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.6.2.195:39293)
0924 10:08:22.224734 (+ 1618us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:22.224740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:22.224996 (+ 256us) server_negotiation.cc:407] Connection header received
0924 10:08:22.225223 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:22.225230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:22.225321 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:22.225428 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:08:22.225877 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.227142 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.229117 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.229917 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.230048 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:22.230590 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:08:22.230781 (+ 191us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:08:22.230886 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:22.231050 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 10:08:22.231151 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1454,"thread_start_us":1289,"threads_started":1}
I20260924 10:08:22.234489 14068 raft_consensus.cc:1261] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Refusing update from remote peer 6c8d7c65d75b443c81a6a512ded1d1b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:22.234647 13924 raft_consensus.cc:1261] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Refusing update from remote peer 6c8d7c65d75b443c81a6a512ded1d1b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:22.235260 14283 consensus_queue.cc:1035] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:22.235713 14283 consensus_queue.cc:1035] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:22.258642 13765 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:22.258898 13765 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:22.262364 13765 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b: Sending DeleteTablet for 3 replicas of tablet d435561f445240bdb5866d166426b90a
I20260924 10:08:22.263219 13903 tablet_service.cc:1514] Processing DeleteTablet for tablet d435561f445240bdb5866d166426b90a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:22 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:39804
I20260924 10:08:22.263578 14300 tablet_replica.cc:330] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5: stopping tablet replica
I20260924 10:08:22.264007 14043 tablet_service.cc:1514] Processing DeleteTablet for tablet d435561f445240bdb5866d166426b90a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:22 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:36054
I20260924 10:08:22.264317 14299 tablet_replica.cc:330] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044: stopping tablet replica
I20260924 10:08:22.264496 14299 raft_consensus.cc:2229] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:22.264712 14299 raft_consensus.cc:2258] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:22.266060 14299 ts_tablet_manager.cc:1905] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:22.272187 14182 tablet_service.cc:1514] Processing DeleteTablet for tablet d435561f445240bdb5866d166426b90a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:22 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:52100
I20260924 10:08:22.272467 14301 tablet_replica.cc:330] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1: stopping tablet replica
I20260924 10:08:22.272634 14301 raft_consensus.cc:2229] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:08:22.272928 14301 raft_consensus.cc:2258] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:22.274266 14301 ts_tablet_manager.cc:1905] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:22.278342 13765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 10:08:22.279497 14299 ts_tablet_manager.cc:1918] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:22.279675 14299 log.cc:1196] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/d435561f445240bdb5866d166426b90a
I20260924 10:08:22.280203 14299 ts_tablet_manager.cc:1939] T d435561f445240bdb5866d166426b90a P ca4d5fa0bfa64ca0a5254009350e3044: Deleting consensus metadata
W20260924 10:08:22.280460 13765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:08:22.281495 13754 catalog_manager.cc:4841] TS ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194:44605): tablet d435561f445240bdb5866d166426b90a (table test-table [id=707d56b67fdf47d8b22f0656a23df686]) successfully deleted
I20260924 10:08:22.288697 14300 raft_consensus.cc:2229] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:22.289274 14300 pending_rounds.cc:62] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5: Trying to abort 2 pending ops.
I20260924 10:08:22.289388 14300 pending_rounds.cc:68] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7332841481146515456 op_type: WRITE_OP write_request { tablet_id: "d435561f445240bdb5866d166426b90a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\302\357\323\325\006\"8\n\ntest-admin\022*\n 707d56b67fdf47d8b22f0656a23df686\020\001\030\001 \001(\001" signature: "<redacted>""\343\177\375\277\366\035\304:\203\315\241dv\0100\357%\300\347#r\rN\005^\200\360\376jR\323r\277\251~KS\310\333\374\372\303gJ)\021\360\342\324\342/r\316rEOL\372\213\223\205&\256_" signing_key_seq_num: 0 } } request_id { client_id: "703c05116a3c4f3292b89a11ee81175d" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260924 10:08:22.289999 14300 pending_rounds.cc:68] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332841480786481152 op_type: NO_OP noop_request { }
I20260924 10:08:22.290143 14300 raft_consensus.cc:2866] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 10:08:22.290297 14300 raft_consensus.cc:2258] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:22.291296 14300 ts_tablet_manager.cc:1905] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:22.291343 14301 ts_tablet_manager.cc:1918] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:22.291469 14301 log.cc:1196] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/d435561f445240bdb5866d166426b90a
I20260924 10:08:22.291904 14301 ts_tablet_manager.cc:1939] T d435561f445240bdb5866d166426b90a P 6c8d7c65d75b443c81a6a512ded1d1b1: Deleting consensus metadata
I20260924 10:08:22.293007 13903 tablet_service.cc:1467] Processing CreateTablet for tablet f1db7ef15f574e6b87640bbf2845dd74 (DEFAULT_TABLE table=test-table [id=609eb4d93029489c85cd54d14d1af303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:22.293423 13903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1db7ef15f574e6b87640bbf2845dd74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:22.296252 14182 tablet_service.cc:1467] Processing CreateTablet for tablet f1db7ef15f574e6b87640bbf2845dd74 (DEFAULT_TABLE table=test-table [id=609eb4d93029489c85cd54d14d1af303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:22.296681 14182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1db7ef15f574e6b87640bbf2845dd74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:22.297763 14300 ts_tablet_manager.cc:1918] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:22.297894 14300 log.cc:1196] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/d435561f445240bdb5866d166426b90a
I20260924 10:08:22.298295 14300 ts_tablet_manager.cc:1939] T d435561f445240bdb5866d166426b90a P f86812a9280f479bbcce24732391b5c5: Deleting consensus metadata
I20260924 10:08:22.299539 13752 catalog_manager.cc:4841] TS f86812a9280f479bbcce24732391b5c5 (127.6.2.193:41261): tablet d435561f445240bdb5866d166426b90a (table test-table [id=707d56b67fdf47d8b22f0656a23df686]) successfully deleted
I20260924 10:08:22.299767 13752 catalog_manager.cc:4841] TS 6c8d7c65d75b443c81a6a512ded1d1b1 (127.6.2.195:39293): tablet d435561f445240bdb5866d166426b90a (table test-table [id=707d56b67fdf47d8b22f0656a23df686]) successfully deleted
I20260924 10:08:22.299870 14043 tablet_service.cc:1467] Processing CreateTablet for tablet f1db7ef15f574e6b87640bbf2845dd74 (DEFAULT_TABLE table=test-table [id=609eb4d93029489c85cd54d14d1af303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:22.300302 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1db7ef15f574e6b87640bbf2845dd74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:22.303864 14277 tablet_bootstrap.cc:492] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5: Bootstrap starting.
I20260924 10:08:22.305512 14277 tablet_bootstrap.cc:654] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:22.308153 14276 tablet_bootstrap.cc:492] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044: Bootstrap starting.
I20260924 10:08:22.309994 14276 tablet_bootstrap.cc:654] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:22.312502 14279 tablet_bootstrap.cc:492] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1: Bootstrap starting.
I20260924 10:08:22.314713 14279 tablet_bootstrap.cc:654] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:22.319415 14276 tablet_bootstrap.cc:492] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044: No bootstrap required, opened a new log
I20260924 10:08:22.319604 14276 ts_tablet_manager.cc:1397] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.005s
I20260924 10:08:22.320590 14276 raft_consensus.cc:348] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:22.320885 14276 raft_consensus.cc:374] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:22.320976 14276 raft_consensus.cc:729] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4d5fa0bfa64ca0a5254009350e3044, State: Initialized, Role: FOLLOWER
I20260924 10:08:22.321354 14276 consensus_queue.cc:260] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:22.325040 14276 ts_tablet_manager.cc:1428] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:08:22.327741 14277 tablet_bootstrap.cc:492] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5: No bootstrap required, opened a new log
I20260924 10:08:22.327967 14277 ts_tablet_manager.cc:1397] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.008s
I20260924 10:08:22.328830 14277 raft_consensus.cc:348] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:22.329066 14277 raft_consensus.cc:374] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:22.329162 14277 raft_consensus.cc:729] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86812a9280f479bbcce24732391b5c5, State: Initialized, Role: FOLLOWER
I20260924 10:08:22.329290 14279 tablet_bootstrap.cc:492] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1: No bootstrap required, opened a new log
I20260924 10:08:22.329464 14279 ts_tablet_manager.cc:1397] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.010s
I20260924 10:08:22.329420 14277 consensus_queue.cc:260] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:22.330017 14277 ts_tablet_manager.cc:1428] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 10:08:22.330289 14279 raft_consensus.cc:348] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:22.330569 14279 raft_consensus.cc:374] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:22.330659 14279 raft_consensus.cc:729] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8d7c65d75b443c81a6a512ded1d1b1, State: Initialized, Role: FOLLOWER
I20260924 10:08:22.330876 14279 consensus_queue.cc:260] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:22.331357 14279 ts_tablet_manager.cc:1428] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:08:22.471375 14284 raft_consensus.cc:482] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:08:22.471581 14284 raft_consensus.cc:504] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:22.473089 14284 leader_election.cc:288] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194:44605), 6c8d7c65d75b443c81a6a512ded1d1b1 (127.6.2.195:39293)
I20260924 10:08:22.481773 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:22.473392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:56433 (local address 127.6.2.194:44605)
0924 10:08:22.474616 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:22.474620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:22.474636 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:08:22.474695 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:22.474698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:22.474811 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:22.474906 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:22.475523 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.476561 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.479420 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.480724 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.480854 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:22.480963 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:22.481324 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 10:08:22.481466 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":1004,"threads_started":1}
I20260924 10:08:22.481739 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:22.473794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:43075 (local address 127.6.2.195:39293)
0924 10:08:22.473986 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:22.474026 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:22.474326 (+ 300us) server_negotiation.cc:407] Connection header received
0924 10:08:22.474471 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:22.474476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:22.474572 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:22.474665 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:22.475285 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.476437 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.479417 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.480706 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.480854 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:22.480964 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:22.481324 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 10:08:22.481463 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 10:08:22.482060 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:22.473402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:39293 (local address 127.6.2.193:43075)
0924 10:08:22.474244 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:22.474260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:22.474359 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:22.474693 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:22.474697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:22.474725 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:22.475135 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:22.475153 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.476579 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.476583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:22.479238 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:22.479255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.480907 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.480919 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:22.481130 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:22.481181 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 10:08:22.481283 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 10:08:22.481505 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":203,"thread_start_us":631,"threads_started":1}
I20260924 10:08:22.482429 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:22.473273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:44605 (local address 127.6.2.193:56433)
0924 10:08:22.473981 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:22.474007 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:22.474159 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:22.474948 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:22.474952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:22.474973 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:22.475375 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:22.475384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.476685 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.476690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:22.479238 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:22.479255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.480933 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.480939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:22.481123 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:22.481198 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 10:08:22.481283 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 10:08:22.481505 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":201,"thread_start_us":391,"threads_started":1}
I20260924 10:08:22.482836 14202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1db7ef15f574e6b87640bbf2845dd74" candidate_uuid: "f86812a9280f479bbcce24732391b5c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" is_pre_election: true
I20260924 10:08:22.483000 14068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1db7ef15f574e6b87640bbf2845dd74" candidate_uuid: "f86812a9280f479bbcce24732391b5c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" is_pre_election: true
I20260924 10:08:22.483130 14202 raft_consensus.cc:2454] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f86812a9280f479bbcce24732391b5c5 in term 0.
I20260924 10:08:22.483279 14068 raft_consensus.cc:2454] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f86812a9280f479bbcce24732391b5c5 in term 0.
I20260924 10:08:22.483678 13855 leader_election.cc:302] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c8d7c65d75b443c81a6a512ded1d1b1, f86812a9280f479bbcce24732391b5c5; no voters:
I20260924 10:08:22.484135 14284 raft_consensus.cc:2781] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:08:22.484261 14284 raft_consensus.cc:482] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:08:22.484355 14284 raft_consensus.cc:3037] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:22.486796 14284 raft_consensus.cc:504] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:22.487471 14284 leader_election.cc:288] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [CANDIDATE]: Term 1 election: Requested vote from peers ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194:44605), 6c8d7c65d75b443c81a6a512ded1d1b1 (127.6.2.195:39293)
I20260924 10:08:22.487989 14068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1db7ef15f574e6b87640bbf2845dd74" candidate_uuid: "f86812a9280f479bbcce24732391b5c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4d5fa0bfa64ca0a5254009350e3044"
I20260924 10:08:22.488035 14202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1db7ef15f574e6b87640bbf2845dd74" candidate_uuid: "f86812a9280f479bbcce24732391b5c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1"
I20260924 10:08:22.488216 14068 raft_consensus.cc:3037] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:22.488234 14202 raft_consensus.cc:3037] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:22.491204 14202 raft_consensus.cc:2454] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f86812a9280f479bbcce24732391b5c5 in term 1.
I20260924 10:08:22.491204 14068 raft_consensus.cc:2454] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f86812a9280f479bbcce24732391b5c5 in term 1.
I20260924 10:08:22.491659 13857 leader_election.cc:302] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca4d5fa0bfa64ca0a5254009350e3044, f86812a9280f479bbcce24732391b5c5; no voters:
I20260924 10:08:22.492024 14284 raft_consensus.cc:2781] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:22.492436 14284 raft_consensus.cc:686] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [term 1 LEADER]: Becoming Leader. State: Replica: f86812a9280f479bbcce24732391b5c5, State: Running, Role: LEADER
I20260924 10:08:22.492851 14284 consensus_queue.cc:237] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } }
I20260924 10:08:22.496290 13766 catalog_manager.cc:5495] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f86812a9280f479bbcce24732391b5c5 (127.6.2.193). New cstate: current_term: 1 leader_uuid: "f86812a9280f479bbcce24732391b5c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f86812a9280f479bbcce24732391b5c5" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 41261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 } health_report { overall_health: UNKNOWN } } }
I20260924 10:08:22.556064 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:22.550200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.6.2.193:41261)
0924 10:08:22.550844 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:22.550848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:22.550870 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:08:22.550954 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:22.550957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:22.551033 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:22.551235 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:08:22.551616 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.552400 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.553697 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.554169 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.554269 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:22.554718 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:08:22.555378 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:08:22.555497 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:22.555616 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 10:08:22.555687 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":425,"threads_started":1}
I20260924 10:08:22.571516 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:22.565787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:46339 (local address 127.6.2.195:39293)
0924 10:08:22.565992 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:22.565998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:22.566323 (+ 325us) server_negotiation.cc:407] Connection header received
0924 10:08:22.566554 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:22.566558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:22.566626 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:22.566726 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:08:22.567382 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.568177 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.569616 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.570067 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.570230 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:22.570670 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:08:22.570779 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:08:22.570881 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:22.571019 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 10:08:22.571140 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:08:22.571607 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:22.565402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:39293 (local address 127.6.2.193:46339)
0924 10:08:22.566213 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:22.566239 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:22.566404 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:22.566748 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:22.566753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:22.566765 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:08:22.567180 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:22.567207 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.568317 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.568323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:22.569469 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:22.569481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.570254 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.570261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:22.570419 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:22.570473 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:08:22.570881 (+ 408us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:08:22.570895 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 10:08:22.570999 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 10:08:22.571165 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"spinlock_wait_cycles":79616,"thread_start_us":387,"threads_started":1}
I20260924 10:08:22.579747 14202 raft_consensus.cc:1261] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Refusing update from remote peer f86812a9280f479bbcce24732391b5c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:22.579874 14068 raft_consensus.cc:1261] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Refusing update from remote peer f86812a9280f479bbcce24732391b5c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:22.580456 14360 consensus_queue.cc:1035] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c8d7c65d75b443c81a6a512ded1d1b1" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 39293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:22.580852 14284 consensus_queue.cc:1035] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca4d5fa0bfa64ca0a5254009350e3044" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:22.609817 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:22.604023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:35709 (local address 127.6.2.193:41261)
0924 10:08:22.604185 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:22.604192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:22.604914 (+ 722us) server_negotiation.cc:407] Connection header received
0924 10:08:22.605154 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:22.605160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:22.605241 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:22.605339 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:08:22.605771 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.606558 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.608054 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.608507 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.608657 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:22.609032 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:08:22.609144 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:08:22.609247 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:22.609391 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 10:08:22.609470 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 10:08:22.609929 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:22.603972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:41261 (local address 127.6.2.195:35709)
0924 10:08:22.604813 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:22.604829 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:22.604983 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:22.605357 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:22.605362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:22.605374 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:08:22.605628 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:22.605637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.606702 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.606708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:22.607907 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:22.607920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.608671 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.608677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:22.608834 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:22.608872 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:08:22.609248 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:08:22.609263 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 10:08:22.609366 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 10:08:22.609505 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":475,"threads_started":1}
I20260924 10:08:22.639288 13766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:22.639633 13766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:22.642712 13766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32c9b81de2f14df98bbc6eb56f282e0b: Sending DeleteTablet for 3 replicas of tablet f1db7ef15f574e6b87640bbf2845dd74
I20260924 10:08:22.643625 14043 tablet_service.cc:1514] Processing DeleteTablet for tablet f1db7ef15f574e6b87640bbf2845dd74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:22 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:36054
I20260924 10:08:22.643743 13903 tablet_service.cc:1514] Processing DeleteTablet for tablet f1db7ef15f574e6b87640bbf2845dd74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:22 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:39804
I20260924 10:08:22.643821 14182 tablet_service.cc:1514] Processing DeleteTablet for tablet f1db7ef15f574e6b87640bbf2845dd74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:22 UTC) from {username='oryx', principal='oryx/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:52100
I20260924 10:08:22.643976 14299 tablet_replica.cc:330] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044: stopping tablet replica
I20260924 10:08:22.644008 14300 tablet_replica.cc:330] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5: stopping tablet replica
I20260924 10:08:22.644100 14301 tablet_replica.cc:330] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1: stopping tablet replica
I20260924 10:08:22.644207 14299 raft_consensus.cc:2229] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:22.644222 14300 raft_consensus.cc:2229] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:08:22.644294 14301 raft_consensus.cc:2229] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:22.644444 14299 raft_consensus.cc:2258] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:22.644469 14301 raft_consensus.cc:2258] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:22.644783 14300 raft_consensus.cc:2258] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:22.645674 14301 ts_tablet_manager.cc:1905] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:22.645896 14299 ts_tablet_manager.cc:1905] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:22.647317 14300 ts_tablet_manager.cc:1905] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:22.651515 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 13838
I20260924 10:08:22.652266 14301 ts_tablet_manager.cc:1918] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:08:22.652421 14301 log.cc:1196] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/f1db7ef15f574e6b87640bbf2845dd74
I20260924 10:08:22.652773 14299 ts_tablet_manager.cc:1918] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:08:22.652880 14301 ts_tablet_manager.cc:1939] T f1db7ef15f574e6b87640bbf2845dd74 P 6c8d7c65d75b443c81a6a512ded1d1b1: Deleting consensus metadata
I20260924 10:08:22.653048 14299 log.cc:1196] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/f1db7ef15f574e6b87640bbf2845dd74
I20260924 10:08:22.653551 14299 ts_tablet_manager.cc:1939] T f1db7ef15f574e6b87640bbf2845dd74 P ca4d5fa0bfa64ca0a5254009350e3044: Deleting consensus metadata
I20260924 10:08:22.654350 13752 catalog_manager.cc:4841] TS 6c8d7c65d75b443c81a6a512ded1d1b1 (127.6.2.195:39293): tablet f1db7ef15f574e6b87640bbf2845dd74 (table test-table [id=609eb4d93029489c85cd54d14d1af303]) successfully deleted
I20260924 10:08:22.654647 13754 catalog_manager.cc:4841] TS ca4d5fa0bfa64ca0a5254009350e3044 (127.6.2.194:44605): tablet f1db7ef15f574e6b87640bbf2845dd74 (table test-table [id=609eb4d93029489c85cd54d14d1af303]) successfully deleted
I20260924 10:08:22.659685 14300 ts_tablet_manager.cc:1918] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:08:22.659906 14300 log.cc:1196] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/f1db7ef15f574e6b87640bbf2845dd74
I20260924 10:08:22.660384 14300 ts_tablet_manager.cc:1939] T f1db7ef15f574e6b87640bbf2845dd74 P f86812a9280f479bbcce24732391b5c5: Deleting consensus metadata
I20260924 10:08:22.661273 13752 catalog_manager.cc:4841] TS f86812a9280f479bbcce24732391b5c5 (127.6.2.193:41261): tablet f1db7ef15f574e6b87640bbf2845dd74 (table test-table [id=609eb4d93029489c85cd54d14d1af303]) successfully deleted
Sep 24 10:08:22 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.2.193@KRBTEST.COM
I20260924 10:08:22.675572 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:22.653134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.6.2.193:41261)
0924 10:08:22.653315 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:22.653321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:22.659602 (+ 6281us) server_negotiation.cc:407] Connection header received
0924 10:08:22.659828 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:22.659833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:22.659930 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:22.660064 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:22.661586 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.662310 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.664168 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.664618 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.664725 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:22.669536 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:22.669552 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:22.669567 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:22.669607 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:22.672835 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:22.673517 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:22.673524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:22.673526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:22.673653 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:22.674020 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:22.674030 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:22.674032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:22.674391 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:22.674736 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:22.674959 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 10:08:22.675212 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:08:22.676283 13969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:22.788429 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 13838
I20260924 10:08:22.809864 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 13978
Sep 24 10:08:22 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.2.194@KRBTEST.COM
I20260924 10:08:22.824540 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:22.810671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.6.2.194:44605)
0924 10:08:22.810852 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:22.810856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:22.810873 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:08:22.811044 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:22.811048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:22.811194 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:22.811298 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:22.812554 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.813265 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.814459 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.814863 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.814960 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:22.819052 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:22.819118 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:22.819134 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:22.819172 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:22.821847 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:22.822500 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:22.822507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:22.822510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:22.822649 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:22.823016 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:22.823022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:22.823025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:22.823399 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:22.823696 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:22.824010 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 10:08:22.824231 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:08:22.825182 14108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:22.949187 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 13978
I20260924 10:08:22.971167 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 14117
Sep 24 10:08:22 dist-test-slave-s3n3 krb5kdc[13722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.2.195@KRBTEST.COM
I20260924 10:08:22.986893 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:22.971958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.6.2.195:39293)
0924 10:08:22.972157 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:22.972161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:22.972181 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:08:22.972360 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:22.972364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:22.972424 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:22.972509 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:22.973721 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.974414 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.975920 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:22.976355 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:22.976453 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:22.981149 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:22.981163 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:22.981180 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:22.981225 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:22.984095 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:22.984923 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:22.984930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:22.984932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:22.985052 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:22.985370 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:22.985373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:22.985375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:22.985726 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:22.985954 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:22.986345 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 10:08:22.986625 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:08:22.987673 14247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:23.134097 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 14117
I20260924 10:08:23.159296 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 13735
I20260924 10:08:23.160506 13807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:23.277565 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 13735
2026-09-24T10:08:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5874 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:08:23 dist-test-slave-s3n3 krb5kdc[14381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:08:23 dist-test-slave-s3n3 krb5kdc[14381](info): set up 2 sockets
Sep 24 10:08:23 dist-test-slave-s3n3 krb5kdc[14381](info): commencing operation
krb5kdc: starting...
W20260924 10:08:24.408653 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.007s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:08:24 dist-test-slave-s3n3 krb5kdc[14381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:08:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:08:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:08:24.646697 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34151
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:46311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:34151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:24.976630 14394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:24.976995 14394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:24.977162 14394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:24.984484 14394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:24.984660 14394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:24.984772 14394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:24.984880 14394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:24.985052 14394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:08:24.995740 14394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:46311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:34151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:24.997159 14394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:24.998528 14394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:25.010568 14399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:25.010473 14400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:25.010493 14402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:25.011185 14394 server_base.cc:1034] running on GCE node
I20260924 10:08:25.012045 14394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:25.013496 14394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:25.014717 14394 hybrid_clock.cc:648] HybridClock initialized: now 1790244505014655 us; error 58 us; skew 500 ppm
I20260924 10:08:25.015025 14394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T10:08:25Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1810 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:08:25 dist-test-slave-s3n3 krb5kdc[14408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:08:25 dist-test-slave-s3n3 krb5kdc[14408](info): set up 2 sockets
Sep 24 10:08:25 dist-test-slave-s3n3 krb5kdc[14408](info): commencing operation
krb5kdc: starting...
W20260924 10:08:26.200160 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:08:26 dist-test-slave-s3n3 krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:08:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:08:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:08:26.431455 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34999
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:34999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:26.762477 14421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:26.762780 14421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:26.762928 14421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:26.770409 14421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:26.770530 14421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:26.770576 14421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:26.770671 14421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:26.770757 14421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:08:26.781100 14421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:34999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:26.782573 14421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:26.783931 14421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:26.796375 14427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:26.796397 14426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:26.796440 14429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:26.797369 14421 server_base.cc:1034] running on GCE node
I20260924 10:08:26.798149 14421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:26.799659 14421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:26.800891 14421 hybrid_clock.cc:648] HybridClock initialized: now 1790244506800847 us; error 32 us; skew 500 ppm
I20260924 10:08:26.801252 14421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:26 dist-test-slave-s3n3 krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244506, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:26.811935 14421 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:08:26.813777 14421 webserver.cc:460] Webserver started at http://127.6.2.254:39761/ using document root <none> and password file <none>
I20260924 10:08:26.814530 14421 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:26.814680 14421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:26.815009 14421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:26.818078 14421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "b6fe999b27b247b5bcaf02bf94eebda5"
format_stamp: "Formatted at 2026-09-24 10:08:26 on dist-test-slave-s3n3"
server_key: "0be0cb8f89f757f16e54cde7c6f6f626"
server_key_iv: "85ba9f0ff825caf59c65a5bd2447e48f"
server_key_version: "encryptionkey@0"
I20260924 10:08:26.818856 14421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "b6fe999b27b247b5bcaf02bf94eebda5"
format_stamp: "Formatted at 2026-09-24 10:08:26 on dist-test-slave-s3n3"
server_key: "0be0cb8f89f757f16e54cde7c6f6f626"
server_key_iv: "85ba9f0ff825caf59c65a5bd2447e48f"
server_key_version: "encryptionkey@0"
I20260924 10:08:26.824290 14421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:08:26.828198 14436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:26.829607 14421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:08:26.829805 14421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "b6fe999b27b247b5bcaf02bf94eebda5"
format_stamp: "Formatted at 2026-09-24 10:08:26 on dist-test-slave-s3n3"
server_key: "0be0cb8f89f757f16e54cde7c6f6f626"
server_key_iv: "85ba9f0ff825caf59c65a5bd2447e48f"
server_key_version: "encryptionkey@0"
I20260924 10:08:26.830029 14421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:26.849366 14421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:26.853951 14421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:26.854255 14421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:26.874019 14421 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:34999
I20260924 10:08:26.874040 14487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:34999 every 8 connection(s)
I20260924 10:08:26.875464 14421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:08:26.879228 14488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:26.884311 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 14421
I20260924 10:08:26.884537 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:08:26.885212 6155 external_mini_cluster.cc:1428] Setting key 21cae1a5a3dd7ddb447ee7cdecdcdc0c
I20260924 10:08:26.891489 14488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5: Bootstrap starting.
I20260924 10:08:26.895327 14488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:26.896351 14488 log.cc:824] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:26.899370 14488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5: No bootstrap required, opened a new log
Sep 24 10:08:26 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:08:26.907847 14488 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34999 } }
I20260924 10:08:26.908254 14488 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:26.908442 14488 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6fe999b27b247b5bcaf02bf94eebda5, State: Initialized, Role: FOLLOWER
I20260924 10:08:26.909116 14488 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34999 } }
I20260924 10:08:26.909379 14488 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:08:26.909523 14488 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:08:26.909638 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:26.887312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51272 (local address 127.6.2.254:34999)
0924 10:08:26.888260 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:26.888280 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:26.888335 (+ 55us) server_negotiation.cc:407] Connection header received
0924 10:08:26.890720 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:26.890747 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:26.891304 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:26.891743 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:26.893193 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:26.895159 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:26.896556 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:26.897480 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:26.897699 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:26.901733 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:26.901741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:26.901759 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:26.901806 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:26.905303 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:26.906084 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:26.906091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:26.906097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:26.906296 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:26.906678 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:26.906684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:26.906687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:26.907317 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:26.907608 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:26.908062 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 10:08:26.908509 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":406,"threads_started":1}
I20260924 10:08:26.909730 14488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:26.912837 14488 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34999 } }
I20260924 10:08:26.913404 14488 leader_election.cc:302] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6fe999b27b247b5bcaf02bf94eebda5; no voters:
I20260924 10:08:26.914225 14488 leader_election.cc:288] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:08:26.914413 14493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:26.915194 14493 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 1 LEADER]: Becoming Leader. State: Replica: b6fe999b27b247b5bcaf02bf94eebda5, State: Running, Role: LEADER
I20260924 10:08:26.915764 14493 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34999 } }
I20260924 10:08:26.916462 14488 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:08:26.918753 14494 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34999 } } }
I20260924 10:08:26.919246 14494 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:26.919549 14495 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6fe999b27b247b5bcaf02bf94eebda5. Latest consensus state: current_term: 1 leader_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34999 } } }
I20260924 10:08:26.919986 14495 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:26.921998 14499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:08:26.928205 14499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:08:26.937848 14499 catalog_manager.cc:1347] Generated new cluster ID: 42130f5b79cb432087af567868e51d01
I20260924 10:08:26.938002 14499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:08:26.952167 14499 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:08:26.953572 14499 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:08:26.965930 14499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5: Generated new TSK 0
I20260924 10:08:26.966866 14499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:08:27.026794 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34999
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:27.371832 14514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:27.372239 14514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:27.372417 14514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:27.381671 14514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:27.381788 14514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:27.381984 14514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:08:27.392181 14514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:27.393512 14514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:27.394824 14514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:27.408391 14522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:27.408354 14520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:27.409452 14519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:27.413012 14514 server_base.cc:1034] running on GCE node
I20260924 10:08:27.413803 14514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:27.414991 14514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:27.416273 14514 hybrid_clock.cc:648] HybridClock initialized: now 1790244507416224 us; error 41 us; skew 500 ppm
I20260924 10:08:27.416684 14514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:27 dist-test-slave-s3n3 krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244507, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:27.420276 14514 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:08:27.421898 14514 webserver.cc:460] Webserver started at http://127.6.2.193:39155/ using document root <none> and password file <none>
I20260924 10:08:27.422748 14514 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:27.422890 14514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:27.423332 14514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:27.426343 14514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "58544382322a4646ba11d7bd0bf516ac"
format_stamp: "Formatted at 2026-09-24 10:08:27 on dist-test-slave-s3n3"
server_key: "4a00b22b4aedbf07be3e0ab271c0fe02"
server_key_iv: "83f1c8130da264cf93f63b032b02e893"
server_key_version: "encryptionkey@0"
I20260924 10:08:27.427284 14514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "58544382322a4646ba11d7bd0bf516ac"
format_stamp: "Formatted at 2026-09-24 10:08:27 on dist-test-slave-s3n3"
server_key: "4a00b22b4aedbf07be3e0ab271c0fe02"
server_key_iv: "83f1c8130da264cf93f63b032b02e893"
server_key_version: "encryptionkey@0"
I20260924 10:08:27.432706 14514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:08:27.436462 14529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:27.437877 14514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:08:27.438057 14514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "58544382322a4646ba11d7bd0bf516ac"
format_stamp: "Formatted at 2026-09-24 10:08:27 on dist-test-slave-s3n3"
server_key: "4a00b22b4aedbf07be3e0ab271c0fe02"
server_key_iv: "83f1c8130da264cf93f63b032b02e893"
server_key_version: "encryptionkey@0"
I20260924 10:08:27.438279 14514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:27.458638 14514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:27.463167 14514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:27.463457 14514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:27.464457 14514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:27.466084 14514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:27.466184 14514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:27.466336 14514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:27.466405 14514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:27.500679 14514 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:36629
I20260924 10:08:27.500731 14641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:36629 every 8 connection(s)
I20260924 10:08:27.501950 14514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:08:27.509958 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 14514
I20260924 10:08:27.510174 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:08:27.510836 6155 external_mini_cluster.cc:1428] Setting key 602a980160c7952d941420985bead428
Sep 24 10:08:27 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244507, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:08:27.523626 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:27.504582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:57429 (local address 127.6.2.254:34999)
0924 10:08:27.504994 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:27.504999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:27.505934 (+ 935us) server_negotiation.cc:407] Connection header received
0924 10:08:27.507812 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:27.507817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:27.507909 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:27.508038 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:27.509482 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:27.510273 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:27.512126 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:27.512607 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:27.512765 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:27.516455 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:27.516464 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:27.516471 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:27.516515 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:27.520059 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:27.521056 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:27.521060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:27.521062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:27.521176 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:27.521698 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:27.521702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:27.521704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:27.522027 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:27.522180 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:27.523152 (+ 972us) server_negotiation.cc:299] Negotiation successful
0924 10:08:27.523362 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":195,"threads_started":1}
I20260924 10:08:27.525108 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:27.505092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:57429)
0924 10:08:27.505779 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:27.505816 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:27.507544 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:27.508271 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:27.508279 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:27.508771 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:27.509311 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:27.509328 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:27.510443 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:27.510447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:27.511995 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:27.512009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:27.512887 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:27.512896 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:27.513140 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:27.513904 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:27.513932 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:27.516186 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:27.520225 (+ 4039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:27.520237 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:27.520258 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:27.520921 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:27.521286 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:27.521294 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:27.521300 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:27.521556 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:27.522161 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:27.522173 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:27.522672 (+ 499us) client_negotiation.cc:769] Sending connection context
0924 10:08:27.522972 (+ 300us) client_negotiation.cc:240] Negotiation successful
0924 10:08:27.523618 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":266,"threads_started":1}
I20260924 10:08:27.526875 14642 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34999
I20260924 10:08:27.527228 14642 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:27.528131 14642 heartbeater.cc:511] Master 127.6.2.254:34999 requested a full tablet report, sending...
I20260924 10:08:27.530522 14453 ts_manager.cc:194] Registered new tserver with Master: 58544382322a4646ba11d7bd0bf516ac (127.6.2.193:36629)
I20260924 10:08:27.532871 14453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:57429
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:08:27.566633 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34999
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:27.896996 14648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:27.897282 14648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:27.897467 14648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:27.904831 14648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:27.904953 14648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:27.905140 14648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:08:27.915014 14648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:27.916379 14648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:27.917830 14648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:27.931479 14653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:27.931453 14656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:27.931524 14654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:27.936769 14648 server_base.cc:1034] running on GCE node
I20260924 10:08:27.937418 14648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:27.938453 14648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:27.939656 14648 hybrid_clock.cc:648] HybridClock initialized: now 1790244507939613 us; error 35 us; skew 500 ppm
I20260924 10:08:27.940014 14648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:27 dist-test-slave-s3n3 krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244507, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:27.943462 14648 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:08:27.945144 14648 webserver.cc:460] Webserver started at http://127.6.2.194:45301/ using document root <none> and password file <none>
I20260924 10:08:27.945909 14648 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:27.946046 14648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:27.946373 14648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:27.949379 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "7847090e3db94126887db09e287f3e2e"
format_stamp: "Formatted at 2026-09-24 10:08:27 on dist-test-slave-s3n3"
server_key: "d0d6a3c5aa75ba15813b30a56c69b995"
server_key_iv: "8d3bf8d82a29c6cd519e18a47fcac000"
server_key_version: "encryptionkey@0"
I20260924 10:08:27.950184 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "7847090e3db94126887db09e287f3e2e"
format_stamp: "Formatted at 2026-09-24 10:08:27 on dist-test-slave-s3n3"
server_key: "d0d6a3c5aa75ba15813b30a56c69b995"
server_key_iv: "8d3bf8d82a29c6cd519e18a47fcac000"
server_key_version: "encryptionkey@0"
I20260924 10:08:27.955489 14648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:08:27.959048 14663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:27.960471 14648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:08:27.960681 14648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "7847090e3db94126887db09e287f3e2e"
format_stamp: "Formatted at 2026-09-24 10:08:27 on dist-test-slave-s3n3"
server_key: "d0d6a3c5aa75ba15813b30a56c69b995"
server_key_iv: "8d3bf8d82a29c6cd519e18a47fcac000"
server_key_version: "encryptionkey@0"
I20260924 10:08:27.960891 14648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:27.983774 14648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:27.988371 14648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:27.988757 14648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:27.989751 14648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:27.991098 14648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:27.991209 14648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:27.991357 14648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:27.991429 14648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:28.025094 14648 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:42231
I20260924 10:08:28.025187 14775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:42231 every 8 connection(s)
I20260924 10:08:28.026616 14648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:08:28.028970 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 14648
I20260924 10:08:28.029184 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:08:28.029790 6155 external_mini_cluster.cc:1428] Setting key fafc89ef805f903fab111a8f464393bf
Sep 24 10:08:28 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244507, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:08:28.052426 14778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:28.029323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:59015 (local address 127.6.2.254:34999)
0924 10:08:28.029816 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:28.029824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:28.031136 (+ 1312us) server_negotiation.cc:407] Connection header received
0924 10:08:28.033297 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:28.033301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:28.033365 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:28.033496 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:28.035680 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.036422 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.039252 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.040119 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.040349 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:28.044822 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:28.044832 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:28.044839 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:28.044886 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:28.048640 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:28.049497 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:28.049506 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:28.049513 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:28.049680 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:28.050154 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:28.050162 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:28.050168 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:28.050568 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:28.050745 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:28.051601 (+ 856us) server_negotiation.cc:299] Negotiation successful
0924 10:08:28.051875 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":249,"threads_started":1}
I20260924 10:08:28.053033 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:28.029887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:59015)
0924 10:08:28.030707 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:28.030761 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:28.032998 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:28.033804 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:28.033821 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:28.034626 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:28.035472 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:28.035500 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.036585 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.036593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:28.038841 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:28.038860 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.040418 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.040423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:28.040634 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:28.041380 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:28.041404 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:28.044501 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:28.048815 (+ 4314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:28.048824 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:28.048839 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:28.049343 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:28.049821 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:28.049826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:28.049828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:28.050022 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:28.050746 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:28.050757 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:28.051194 (+ 437us) client_negotiation.cc:769] Sending connection context
0924 10:08:28.051432 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 10:08:28.051854 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":291,"threads_started":1}
I20260924 10:08:28.054698 14776 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34999
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:08:28.055025 14776 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:28.055780 14776 heartbeater.cc:511] Master 127.6.2.254:34999 requested a full tablet report, sending...
I20260924 10:08:28.057334 14453 ts_manager.cc:194] Registered new tserver with Master: 7847090e3db94126887db09e287f3e2e (127.6.2.194:42231)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
I20260924 10:08:28.058349 14453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:59015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:08:28.087594 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34999
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:28.421849 14782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:28.422153 14782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:28.422302 14782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:28.429888 14782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:28.430054 14782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:28.430204 14782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:08:28.441090 14782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:28.442646 14782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:28.444063 14782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:28.458552 14787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:28.458823 14790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:28.458547 14788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:28.459748 14782 server_base.cc:1034] running on GCE node
I20260924 10:08:28.460439 14782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:28.461524 14782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:28.462786 14782 hybrid_clock.cc:648] HybridClock initialized: now 1790244508462744 us; error 47 us; skew 500 ppm
I20260924 10:08:28.463222 14782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:28 dist-test-slave-s3n3 krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244508, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:28.466918 14782 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:08:28.468798 14782 webserver.cc:460] Webserver started at http://127.6.2.195:44881/ using document root <none> and password file <none>
I20260924 10:08:28.469532 14782 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:28.469693 14782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:28.470044 14782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:28.473258 14782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "729a98fd5f3d48eca50f71e399226af8"
format_stamp: "Formatted at 2026-09-24 10:08:28 on dist-test-slave-s3n3"
server_key: "913ab56ca1f6a129a39681e296e5d65b"
server_key_iv: "f48ece8da76bc170b4cfe7cd198c26a0"
server_key_version: "encryptionkey@0"
I20260924 10:08:28.474063 14782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "729a98fd5f3d48eca50f71e399226af8"
format_stamp: "Formatted at 2026-09-24 10:08:28 on dist-test-slave-s3n3"
server_key: "913ab56ca1f6a129a39681e296e5d65b"
server_key_iv: "f48ece8da76bc170b4cfe7cd198c26a0"
server_key_version: "encryptionkey@0"
I20260924 10:08:28.480444 14782 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 10:08:28.484738 14797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:28.486348 14782 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260924 10:08:28.486629 14782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "729a98fd5f3d48eca50f71e399226af8"
format_stamp: "Formatted at 2026-09-24 10:08:28 on dist-test-slave-s3n3"
server_key: "913ab56ca1f6a129a39681e296e5d65b"
server_key_iv: "f48ece8da76bc170b4cfe7cd198c26a0"
server_key_version: "encryptionkey@0"
I20260924 10:08:28.486863 14782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:28.498184 14782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:28.503713 14782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:28.504215 14782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:28.505496 14782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:28.506938 14782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:28.507048 14782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:28.507251 14782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:28.507326 14782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:28.536847 14642 heartbeater.cc:503] Master 127.6.2.254:34999 was elected leader, sending a full tablet report...
I20260924 10:08:28.549690 14782 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:44199
I20260924 10:08:28.549773 14909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:44199 every 8 connection(s)
I20260924 10:08:28.551123 14782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:08:28.553844 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 14782
I20260924 10:08:28.553999 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:08:28.554574 6155 external_mini_cluster.cc:1428] Setting key bb109f468bdc8b0389bcabc8bccffc71
Sep 24 10:08:28 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244508, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:08:28.573107 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:28.553619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:54003 (local address 127.6.2.254:34999)
0924 10:08:28.554205 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:28.554209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:28.555244 (+ 1035us) server_negotiation.cc:407] Connection header received
0924 10:08:28.557674 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:28.557679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:28.557778 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:28.557891 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:28.559542 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.560588 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.562263 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.562656 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.562780 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:28.566588 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:28.566593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:28.566597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:28.566636 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:28.569931 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:28.570865 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:28.570869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:28.570872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:28.570990 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:28.571509 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:28.571512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:28.571514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:28.571784 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:28.571901 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:28.572612 (+ 711us) server_negotiation.cc:299] Negotiation successful
0924 10:08:28.572819 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":342,"threads_started":1}
I20260924 10:08:28.574103 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:28.554244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:54003)
0924 10:08:28.555008 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:28.555051 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:28.557336 (+ 2285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:28.558093 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:28.558102 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:28.558617 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:28.559258 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:28.559278 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.560797 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.560810 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:28.562105 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:28.562122 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.562875 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.562880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:28.563168 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:28.563943 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:28.563968 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:28.566377 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:28.570180 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:28.570190 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:28.570209 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:28.570745 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:28.571193 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:28.571196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:28.571199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:28.571399 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:28.571892 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:28.571899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:28.572238 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 10:08:28.572485 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 10:08:28.572891 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":322,"threads_started":1}
I20260924 10:08:28.575635 14910 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34999
I20260924 10:08:28.575949 14910 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:28.576648 14910 heartbeater.cc:511] Master 127.6.2.254:34999 requested a full tablet report, sending...
I20260924 10:08:28.578437 14453 ts_manager.cc:194] Registered new tserver with Master: 729a98fd5f3d48eca50f71e399226af8 (127.6.2.195:44199)
I20260924 10:08:28.579443 14453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:54003
I20260924 10:08:28.582901 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab.
Entry for principal oryx/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab.
WARNING: no policy specified for oryx/127.6.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab.
Entry for principal oryx/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab.
WARNING: no policy specified for oryx/127.6.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab.
Entry for principal oryx/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab.
I20260924 10:08:28.730347 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 14514
Sep 24 10:08:28 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:08:28.747630 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:28.731544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.6.2.193:36629)
0924 10:08:28.732243 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:28.732251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:28.732303 (+ 52us) server_negotiation.cc:407] Connection header received
0924 10:08:28.732366 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:28.732372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:28.732582 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:28.732719 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:28.734001 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.735219 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.736540 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.737109 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.737226 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:28.741407 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:28.741424 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:28.741448 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:28.741486 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:28.744449 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:28.745154 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:28.745160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:28.745162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:28.745304 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:28.745732 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:28.745736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:28.745738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:28.746065 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:28.746300 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:28.746817 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 10:08:28.747330 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":375,"threads_started":1}
I20260924 10:08:28.748972 14637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:28.834553 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 14514
I20260924 10:08:28.851218 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 14648
Sep 24 10:08:28 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:08:28.867637 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:28.852134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.6.2.194:42231)
0924 10:08:28.852704 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:28.852711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:28.852738 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:08:28.852798 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:28.852804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:28.853026 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:28.853161 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:28.854368 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.855650 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.856839 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.857388 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.857524 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:28.861734 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:28.861746 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:28.861762 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:28.861813 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:28.865035 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:28.865756 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:28.865763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:28.865766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:28.865887 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:28.866208 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:28.866211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:28.866213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:28.866550 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:28.866800 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:28.867138 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 10:08:28.867352 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":300,"threads_started":1}
I20260924 10:08:28.869156 14771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:28.956377 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 14648
I20260924 10:08:28.974188 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 14782
Sep 24 10:08:28 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:08:28.991231 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:28.975386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.6.2.195:44199)
0924 10:08:28.976087 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:28.976093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:28.976123 (+ 30us) server_negotiation.cc:407] Connection header received
0924 10:08:28.976182 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:28.976188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:28.976403 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:28.976536 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:28.977880 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.978948 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.980410 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:28.981068 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:28.981190 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:28.985647 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:28.985659 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:28.985675 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:28.985709 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:28.988397 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:28.989090 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:28.989097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:28.989099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:28.989224 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:28.989600 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:28.989609 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:28.989611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:28.989935 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:28.990253 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:28.990524 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 10:08:28.990786 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":380,"threads_started":1}
I20260924 10:08:28.992707 14905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:29.077389 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 14782
I20260924 10:08:29.093484 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 14421
I20260924 10:08:29.094705 14483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:29.198807 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 14421
I20260924 10:08:29.220135 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34999
--webserver_interface=127.6.2.254
--webserver_port=39761
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:34999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:29.548552 14928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:29.548857 14928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:29.549026 14928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:29.556465 14928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:29.556584 14928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:29.556679 14928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:29.556767 14928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:29.556835 14928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:08:29.567023 14928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:34999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=39761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:29.568425 14928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:29.569880 14928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:29.582240 14934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:29.582264 14936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:29.582901 14933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:29.583309 14928 server_base.cc:1034] running on GCE node
I20260924 10:08:29.584172 14928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:29.585757 14928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:29.586994 14928 hybrid_clock.cc:648] HybridClock initialized: now 1790244509586930 us; error 61 us; skew 500 ppm
I20260924 10:08:29.587373 14928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:29 dist-test-slave-s3n3 krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244509, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:29.590615 14928 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:08:29.592192 14928 webserver.cc:460] Webserver started at http://127.6.2.254:39761/ using document root <none> and password file <none>
I20260924 10:08:29.592979 14928 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:29.593165 14928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:29.598760 14928 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 10:08:29.602661 14943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:29.604162 14928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:08:29.604317 14928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "b6fe999b27b247b5bcaf02bf94eebda5"
format_stamp: "Formatted at 2026-09-24 10:08:26 on dist-test-slave-s3n3"
server_key: "0be0cb8f89f757f16e54cde7c6f6f626"
server_key_iv: "85ba9f0ff825caf59c65a5bd2447e48f"
server_key_version: "encryptionkey@0"
I20260924 10:08:29.605254 14928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:29.625384 14928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:29.630121 14928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:29.630354 14928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:29.649442 14928 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:34999
I20260924 10:08:29.649485 14994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:34999 every 8 connection(s)
I20260924 10:08:29.651113 14928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:08:29.655093 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 14928
I20260924 10:08:29.656018 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:36629
--local_ip_for_outbound_sockets=127.6.2.193
--tserver_master_addrs=127.6.2.254:34999
--webserver_port=39155
--webserver_interface=127.6.2.193
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.2.193
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:08:29.659823 14995 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:08:29.664994 14995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5: Bootstrap starting.
I20260924 10:08:29.684962 14995 log.cc:824] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:29.693981 14995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:08:29.694545 14995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5: Bootstrap complete.
I20260924 10:08:29.703665 14995 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34999 } }
I20260924 10:08:29.704493 14995 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6fe999b27b247b5bcaf02bf94eebda5, State: Initialized, Role: FOLLOWER
I20260924 10:08:29.705140 14995 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34999 } }
I20260924 10:08:29.705384 14995 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:08:29.705500 14995 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:08:29.705688 14995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:08:29.709877 14995 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34999 } }
I20260924 10:08:29.710395 14995 leader_election.cc:302] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6fe999b27b247b5bcaf02bf94eebda5; no voters:
I20260924 10:08:29.710999 14995 leader_election.cc:288] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:08:29.711225 14999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:08:29.712941 14995 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:08:29.718152 14999 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [term 2 LEADER]: Becoming Leader. State: Replica: b6fe999b27b247b5bcaf02bf94eebda5, State: Running, Role: LEADER
I20260924 10:08:29.718968 14999 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34999 } }
I20260924 10:08:29.721896 15000 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34999 } } }
I20260924 10:08:29.722637 15000 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:29.723122 15001 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6fe999b27b247b5bcaf02bf94eebda5. Latest consensus state: current_term: 2 leader_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fe999b27b247b5bcaf02bf94eebda5" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34999 } } }
I20260924 10:08:29.727656 15001 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:29.735083 15010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:08:29.740694 15010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:08:29.742115 15010 catalog_manager.cc:1259] Loaded cluster ID: 42130f5b79cb432087af567868e51d01
I20260924 10:08:29.742226 15010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:08:29.746219 15010 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:08:29.747753 15010 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b6fe999b27b247b5bcaf02bf94eebda5: Loaded TSK: 0
I20260924 10:08:29.749303 15010 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:08:30.007249 14997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:30.007583 14997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:30.007755 14997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:30.015540 14997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:30.015724 14997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:30.015899 14997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:08:30.026765 14997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.193.keytab
--principal=oryx/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:36629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=39155
--tserver_master_addrs=127.6.2.254:34999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:30.028311 14997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:30.029699 14997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:30.044605 15022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:30.044651 15023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:30.045243 15025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:30.045732 14997 server_base.cc:1034] running on GCE node
I20260924 10:08:30.046355 14997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:30.047489 14997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:30.048738 14997 hybrid_clock.cc:648] HybridClock initialized: now 1790244510048693 us; error 32 us; skew 500 ppm
I20260924 10:08:30.049089 14997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244510, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:30.052600 14997 init.cc:377] Logged in from keytab as oryx/127.6.2.193@KRBTEST.COM (short username oryx)
I20260924 10:08:30.054289 14997 webserver.cc:460] Webserver started at http://127.6.2.193:39155/ using document root <none> and password file <none>
I20260924 10:08:30.055019 14997 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:30.055209 14997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:30.060680 14997 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 10:08:30.063479 15032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:30.064903 14997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:08:30.065062 14997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "58544382322a4646ba11d7bd0bf516ac"
format_stamp: "Formatted at 2026-09-24 10:08:27 on dist-test-slave-s3n3"
server_key: "4a00b22b4aedbf07be3e0ab271c0fe02"
server_key_iv: "83f1c8130da264cf93f63b032b02e893"
server_key_version: "encryptionkey@0"
I20260924 10:08:30.066025 14997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:30.097401 14997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:30.101545 14997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:30.101871 14997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:30.102883 14997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:30.104243 14997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:30.104329 14997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:30.104401 14997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:30.104454 14997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:30.138489 14997 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:36629
I20260924 10:08:30.138854 15144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:36629 every 8 connection(s)
I20260924 10:08:30.140067 14997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:08:30.142544 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 14997
I20260924 10:08:30.143345 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:42231
--local_ip_for_outbound_sockets=127.6.2.194
--tserver_master_addrs=127.6.2.254:34999
--webserver_port=45301
--webserver_interface=127.6.2.194
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.2.194
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:30.158357 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:30.143140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:34609)
0924 10:08:30.144205 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:30.144256 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:30.146106 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:30.147458 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:30.147471 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:30.148065 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:30.148566 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:30.148583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.150129 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.150136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:30.151391 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:30.151403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.152392 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.152400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:30.152667 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:30.153355 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:30.153377 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:30.156992 (+ 3615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":592,"threads_started":1}
W20260924 10:08:30.158381 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:30.143185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:34609 (local address 127.6.2.254:34999)
0924 10:08:30.145017 (+ 1832us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:30.145031 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:30.145082 (+ 51us) server_negotiation.cc:407] Connection header received
0924 10:08:30.146616 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:30.146627 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:30.146933 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:30.147329 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:30.148763 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.149957 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.151541 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.152161 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.152366 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:30.157349 (+ 4983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:34609: BlockingRecv error: recv got EOF from 127.6.2.193:34609 (error 108)
Metrics: {"server-negotiator.queue_time_us":1580,"thread_start_us":1385,"threads_started":1}
W20260924 10:08:30.159250 15145 heartbeater.cc:644] Failed to heartbeat to 127.6.2.254:34999 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.2.254:34999: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:30.167359 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:30.159889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:58619)
0924 10:08:30.160076 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:30.160092 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:30.160208 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:30.160502 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:30.160506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:30.160863 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:30.161001 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:30.161009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.161992 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.161997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:30.162859 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:30.162869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.163605 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.163610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:30.163787 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:30.164364 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:30.164380 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:30.167168 (+ 2788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:08:30.167616 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:30.160067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:58619 (local address 127.6.2.254:34999)
0924 10:08:30.160235 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:30.160243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:30.160263 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:08:30.160317 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:30.160329 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:30.160380 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:30.160542 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:30.161133 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.161846 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.162992 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.163449 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.163593 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:30.167401 (+ 3808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:58619: BlockingRecv error: recv got EOF from 127.6.2.193:58619 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:30.178484 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:30.168080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:37553)
0924 10:08:30.168383 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:30.168406 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:30.168517 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:30.168976 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:30.168989 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:30.169473 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:30.169689 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:30.169699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.171397 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.171402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:30.173008 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:30.173024 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.174200 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.174206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:30.174386 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:30.175039 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:30.175105 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:30.178261 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":69}
W20260924 10:08:30.178787 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:30.168307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:37553 (local address 127.6.2.254:34999)
0924 10:08:30.168594 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:30.168600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:30.168617 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:08:30.168686 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:30.168691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:30.168794 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:30.168910 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:30.169860 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.171233 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.173157 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.173866 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.173997 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:30.178508 (+ 4511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:37553: BlockingRecv error: recv got EOF from 127.6.2.193:37553 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 10:08:30.178812 15145 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:08:30.472661 15149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:30.472954 15149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:30.473047 15149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:30.480281 15149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:30.480406 15149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:30.480525 15149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:08:30.490269 15149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.194.keytab
--principal=oryx/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:42231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=45301
--tserver_master_addrs=127.6.2.254:34999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:30.491607 15149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:30.492837 15149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:30.505677 15154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:30.505651 15157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:30.505762 15155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:30.506455 15149 server_base.cc:1034] running on GCE node
I20260924 10:08:30.507004 15149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:30.507980 15149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:30.509181 15149 hybrid_clock.cc:648] HybridClock initialized: now 1790244510509142 us; error 37 us; skew 500 ppm
I20260924 10:08:30.509511 15149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244510, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:30.512801 15149 init.cc:377] Logged in from keytab as oryx/127.6.2.194@KRBTEST.COM (short username oryx)
I20260924 10:08:30.514437 15149 webserver.cc:460] Webserver started at http://127.6.2.194:45301/ using document root <none> and password file <none>
I20260924 10:08:30.515162 15149 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:30.515277 15149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:30.520591 15149 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 10:08:30.523281 15164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:30.524586 15149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:08:30.524767 15149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "7847090e3db94126887db09e287f3e2e"
format_stamp: "Formatted at 2026-09-24 10:08:27 on dist-test-slave-s3n3"
server_key: "d0d6a3c5aa75ba15813b30a56c69b995"
server_key_iv: "8d3bf8d82a29c6cd519e18a47fcac000"
server_key_version: "encryptionkey@0"
I20260924 10:08:30.525736 15149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:30.548305 15149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:30.552942 15149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:30.553262 15149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:30.554316 15149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:30.555749 15149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:30.555837 15149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:30.555990 15149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:30.556080 15149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:30.590813 15149 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:42231
I20260924 10:08:30.590903 15276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:42231 every 8 connection(s)
I20260924 10:08:30.592271 15149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:08:30.596165 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 15149
I20260924 10:08:30.596915 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:44199
--local_ip_for_outbound_sockets=127.6.2.195
--tserver_master_addrs=127.6.2.254:34999
--webserver_port=44881
--webserver_interface=127.6.2.195
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.2.195
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:08:30.608279 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:30.594969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:50471 (local address 127.6.2.254:34999)
0924 10:08:30.595181 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:30.595185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:30.596369 (+ 1184us) server_negotiation.cc:407] Connection header received
0924 10:08:30.598530 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:30.598536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:30.598645 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:30.598761 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:30.600352 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.601058 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.602604 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.603010 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.603195 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:30.608021 (+ 4826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:50471: BlockingRecv error: recv got EOF from 127.6.2.194:50471 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:08:30.609351 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:30.595473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:50471)
0924 10:08:30.596196 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:30.596232 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:30.598234 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:30.599018 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:30.599027 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:30.599605 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:30.600162 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:30.600179 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.601230 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.601241 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:30.602448 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:30.602460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.603309 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.603316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:30.603542 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:30.604239 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:30.604265 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:30.607843 (+ 3578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":321,"threads_started":1}
W20260924 10:08:30.610193 15277 heartbeater.cc:644] Failed to heartbeat to 127.6.2.254:34999 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.2.254:34999: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:30.620595 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:30.610783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:35823)
0924 10:08:30.611006 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:30.611029 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:30.611177 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:30.611513 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:30.611519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:30.612012 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:30.612241 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:30.612256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.613346 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.613353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:30.615189 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:30.615209 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.615912 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.615920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:30.616143 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:30.616792 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:30.616819 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:30.620275 (+ 3456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 10:08:30.620736 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:30.610920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:35823 (local address 127.6.2.254:34999)
0924 10:08:30.611149 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:30.611156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:30.611179 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:08:30.611307 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:30.611311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:30.611370 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:30.611447 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:30.612400 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.613182 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.615346 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.615747 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.615855 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:30.620493 (+ 4638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:35823: BlockingRecv error: recv got EOF from 127.6.2.194:35823 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:30.631826 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:30.621490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:33715)
0924 10:08:30.621679 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:30.621698 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:30.621792 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:30.622080 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:30.622085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:30.622463 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:30.622716 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:30.622727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.623860 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.623866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:30.625330 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:30.625345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.626081 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.626087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:30.626269 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:30.626892 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:30.626914 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:30.631493 (+ 4579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 10:08:30.632092 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:30.621586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:33715 (local address 127.6.2.254:34999)
0924 10:08:30.621728 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:30.621732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:30.621747 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:08:30.621897 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:30.621900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:30.621957 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:30.622038 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:30.622852 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.623731 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.625483 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:30.625929 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:30.626035 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:30.631730 (+ 5695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:33715: BlockingRecv error: recv got EOF from 127.6.2.194:33715 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 10:08:30.632153 15277 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:08:30.945247 15280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:30.945600 15280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:30.945765 15280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:30.953064 15280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:30.953236 15280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:30.953428 15280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:08:30.963615 15280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/krb5kdc/oryx_127.6.2.195.keytab
--principal=oryx/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:44199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=44881
--tserver_master_addrs=127.6.2.254:34999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:30.964998 15280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:30.966272 15280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:30.979839 15285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:30.979875 15286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:30.980348 15288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:30.980820 15280 server_base.cc:1034] running on GCE node
I20260924 10:08:30.981544 15280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:30.982533 15280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:30.983779 15280 hybrid_clock.cc:648] HybridClock initialized: now 1790244510983742 us; error 41 us; skew 500 ppm
I20260924 10:08:30.984148 15280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:30 dist-test-slave-s3n3 krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244510, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:30.987471 15280 init.cc:377] Logged in from keytab as oryx/127.6.2.195@KRBTEST.COM (short username oryx)
I20260924 10:08:30.988925 15280 webserver.cc:460] Webserver started at http://127.6.2.195:44881/ using document root <none> and password file <none>
I20260924 10:08:30.989584 15280 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:30.989696 15280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:30.995009 15280 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:08:30.997872 15295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:30.999414 15280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:08:30.999600 15280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "729a98fd5f3d48eca50f71e399226af8"
format_stamp: "Formatted at 2026-09-24 10:08:28 on dist-test-slave-s3n3"
server_key: "913ab56ca1f6a129a39681e296e5d65b"
server_key_iv: "f48ece8da76bc170b4cfe7cd198c26a0"
server_key_version: "encryptionkey@0"
I20260924 10:08:31.000577 15280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:31.023330 15280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:31.028654 15280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:31.028967 15280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:31.030128 15280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:31.031613 15280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:31.031744 15280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:31.031879 15280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:31.031967 15280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:31.069741 15280 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:44199
I20260924 10:08:31.069792 15407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:44199 every 8 connection(s)
I20260924 10:08:31.071369 15280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:08:31.081018 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 15280
Sep 24 10:08:31 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:31 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:08:31.089870 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:31.074193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:39033 (local address 127.6.2.254:34999)
0924 10:08:31.074396 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:31.074403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:31.075612 (+ 1209us) server_negotiation.cc:407] Connection header received
0924 10:08:31.077393 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:31.077397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:31.077479 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:31.077569 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:31.079100 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.079853 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.081617 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.082132 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.082293 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:31.089641 (+ 7348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:39033: BlockingRecv error: recv got EOF from 127.6.2.195:39033 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:08:31.091229 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:31.074697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:39033)
0924 10:08:31.075453 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:31.075490 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:31.077141 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:31.077805 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:31.077813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:31.078306 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:31.078884 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:31.078902 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.080068 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.080073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:31.081452 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:31.081464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.082387 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.082393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:31.082739 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:31.083601 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:31.083642 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:31.089369 (+ 5727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":351,"threads_started":1}
W20260924 10:08:31.092074 15408 heartbeater.cc:644] Failed to heartbeat to 127.6.2.254:34999 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.2.254:34999: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Sep 24 10:08:31 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:31 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:31.102130 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:31.082226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.6.2.254:34999)
0924 10:08:31.083495 (+ 1269us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:31.083500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:31.083515 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:08:31.083557 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:31.083561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:31.083647 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:31.083744 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:31.086336 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.087425 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.088919 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.089388 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.089532 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:31.093090 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:31.093105 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:31.093129 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:31.093184 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:31.097505 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:31.098482 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:31.098492 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:31.098502 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:31.098784 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:31.099594 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:31.099599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:31.099603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:31.100362 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:31.100742 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:31.101309 (+ 567us) server_negotiation.cc:299] Negotiation successful
0924 10:08:31.101661 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":740,"threads_started":1}
I20260924 10:08:31.103215 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:31.092756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:33075)
0924 10:08:31.092914 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:31.092935 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:31.093040 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:31.093388 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:31.093395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:31.093821 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:31.094019 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:31.094028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.095869 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.095875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:31.097347 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:31.097366 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.098498 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.098504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:31.098705 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:31.099362 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:31.099379 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:31.102925 (+ 3546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 10:08:31.103720 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:31.092849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:33075 (local address 127.6.2.254:34999)
0924 10:08:31.093011 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:31.093015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:31.093030 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:08:31.093179 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:31.093183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:31.093239 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:31.093319 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:31.094213 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.095717 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.097554 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.098291 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.098442 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:31.103401 (+ 4959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:33075: BlockingRecv error: recv got EOF from 127.6.2.195:33075 (error 108)
Metrics: {"server-negotiator.queue_time_us":65,"spinlock_wait_cycles":5888}
Sep 24 10:08:31 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:31 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:31.112815 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:31.103984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:42215)
0924 10:08:31.104830 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:31.104848 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:31.104953 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:31.105328 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:31.105331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:31.105742 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:31.105935 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:31.105946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.107217 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.107222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:31.108147 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:31.108157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.108924 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.108930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:31.109122 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:31.109724 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:31.109741 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:31.112599 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 10:08:31.113253 15408 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:08:31.113281 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:31.104338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:42215 (local address 127.6.2.254:34999)
0924 10:08:31.104657 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:31.104662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:31.104927 (+ 265us) server_negotiation.cc:407] Connection header received
0924 10:08:31.105039 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:31.105044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:31.105142 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:31.105317 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:31.106078 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.107033 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.108296 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.108774 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.108901 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:31.112866 (+ 3965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:42215: BlockingRecv error: recv got EOF from 127.6.2.195:42215 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
Sep 24 10:08:31 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:31 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:31.187994 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:31.179601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:35685)
0924 10:08:31.180157 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:31.180173 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:31.180336 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:31.180726 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:31.180733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:31.181191 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:31.181378 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:31.181388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.182391 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.182396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:31.183416 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:31.183428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.184168 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.184174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:31.184327 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:31.184917 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:31.184934 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:31.187666 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":297,"threads_started":1}
W20260924 10:08:31.188092 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:31.179640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:35685 (local address 127.6.2.254:34999)
0924 10:08:31.179857 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:31.179864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:31.180255 (+ 391us) server_negotiation.cc:407] Connection header received
0924 10:08:31.180471 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:31.180476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:31.180569 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:31.180672 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:31.181529 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.182246 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.183591 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.183977 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.184109 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:31.187879 (+ 3770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:35685: BlockingRecv error: recv got EOF from 127.6.2.193:35685 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 10:08:31 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:31 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:31.643368 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:31.633042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:47413)
0924 10:08:31.633707 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:31.633729 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:31.634031 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:31.634409 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:31.634414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:31.634904 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:31.635128 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:31.635138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.636287 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.636292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:31.637327 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:31.637340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.638647 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.638654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:31.638873 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:31.639757 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:31.639784 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:31.643125 (+ 3341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":310,"threads_started":1}
W20260924 10:08:31.643936 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:31.633314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:47413 (local address 127.6.2.254:34999)
0924 10:08:31.633635 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:31.633649 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:31.633833 (+ 184us) server_negotiation.cc:407] Connection header received
0924 10:08:31.634171 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:31.634175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:31.634239 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:31.634334 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:31.635274 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.636016 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.637622 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:31.638413 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:31.638544 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:31.643367 (+ 4823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:47413: BlockingRecv error: recv got EOF from 127.6.2.194:47413 (error 108)
Metrics: {"server-negotiator.queue_time_us":173}
Sep 24 10:08:32 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:32 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:32.131909 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:32.114106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:53089)
0924 10:08:32.114665 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:32.114683 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:32.114805 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:32.115179 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:32.115183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:32.115774 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:32.115963 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:32.115973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:32.117131 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:32.117135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:32.118142 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:32.118154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:32.119519 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:32.119526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:32.119772 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:32.127709 (+ 7937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:32.127733 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:32.131580 (+ 3847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":263,"threads_started":1}
W20260924 10:08:32.132350 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:32.114308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:53089 (local address 127.6.2.254:34999)
0924 10:08:32.114563 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:32.114570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:32.114768 (+ 198us) server_negotiation.cc:407] Connection header received
0924 10:08:32.114910 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:32.114914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:32.114977 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:32.115120 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:32.116120 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:32.116965 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:32.118392 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:32.119132 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:32.119404 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:32.131818 (+ 12414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:53089: BlockingRecv error: recv got EOF from 127.6.2.195:53089 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 24 10:08:32 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:32 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:32.197885 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:32.189073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:53739)
0924 10:08:32.189651 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:32.189666 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:32.189830 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:32.190199 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:32.190204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:32.190650 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:32.190786 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:32.190794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:32.191882 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:32.191887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:32.192770 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:32.192780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:32.193587 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:32.193596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:32.193807 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:32.194539 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:32.194563 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:32.197641 (+ 3078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":280,"threads_started":1}
W20260924 10:08:32.198196 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:32.189159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:53739 (local address 127.6.2.254:34999)
0924 10:08:32.189410 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:32.189418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:32.189753 (+ 335us) server_negotiation.cc:407] Connection header received
0924 10:08:32.189956 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:32.189961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:32.190029 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:32.190167 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:32.190929 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:32.191740 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:32.192908 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:32.193343 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:32.193472 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:32.197841 (+ 4369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:53739: BlockingRecv error: recv got EOF from 127.6.2.193:53739 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 24 10:08:32 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:32 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:32.653829 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:32.644481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:33275)
0924 10:08:32.645055 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:32.645071 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:32.645182 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:32.645628 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:32.645635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:32.646058 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:32.646201 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:32.646209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:32.647269 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:32.647273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:32.648198 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:32.648210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:32.649015 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:32.649020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:32.649185 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:32.649893 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:32.649912 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:32.653488 (+ 3576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":324,"threads_started":1}
W20260924 10:08:32.654026 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:32.644567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:33275 (local address 127.6.2.254:34999)
0924 10:08:32.644743 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:32.644749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:32.645138 (+ 389us) server_negotiation.cc:407] Connection header received
0924 10:08:32.645323 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:32.645328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:32.645398 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:32.645570 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:32.646369 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:32.647132 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:32.648347 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:32.648808 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:32.648914 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:32.653770 (+ 4856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:33275: BlockingRecv error: recv got EOF from 127.6.2.194:33275 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 10:08:33 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:33 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:33.143184 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:33.133240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:57251)
0924 10:08:33.134121 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:33.134142 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:33.134294 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:33.134745 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:33.134752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:33.135485 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:33.135678 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:33.135690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:33.136731 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:33.136736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:33.137683 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:33.137696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:33.138823 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:33.138829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:33.138986 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:33.139746 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:33.139769 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:33.142861 (+ 3092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":386,"threads_started":1}
W20260924 10:08:33.143436 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:33.133438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:57251 (local address 127.6.2.254:34999)
0924 10:08:33.133654 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:33.133659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:33.134249 (+ 590us) server_negotiation.cc:407] Connection header received
0924 10:08:33.134427 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:33.134431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:33.134511 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:33.134607 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:33.135848 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:33.136581 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:33.137897 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:33.138660 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:33.138769 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:33.143134 (+ 4365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:57251: BlockingRecv error: recv got EOF from 127.6.2.195:57251 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 24 10:08:33 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:33 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:33.208796 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:33.198861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:60911)
0924 10:08:33.199467 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:33.199483 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:33.199653 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:33.200036 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:33.200041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:33.200675 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:33.201119 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:33.201137 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:33.202235 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:33.202243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:33.203229 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:33.203240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:33.204129 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:33.204135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:33.204317 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:33.204964 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:33.204981 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:33.208368 (+ 3387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":371,"threads_started":1}
W20260924 10:08:33.208827 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:33.199101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:60911 (local address 127.6.2.254:34999)
0924 10:08:33.199362 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:33.199368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:33.199571 (+ 203us) server_negotiation.cc:407] Connection header received
0924 10:08:33.199819 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:33.199824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:33.199896 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:33.200006 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:33.201305 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:33.202046 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:33.203369 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:33.203969 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:33.204112 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:33.208578 (+ 4466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:60911: BlockingRecv error: recv got EOF from 127.6.2.193:60911 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 10:08:33 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:33 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:33.665160 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:33.655247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:55881)
0924 10:08:33.655767 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:33.655784 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:33.655960 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:33.656311 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:33.656316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:33.656685 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:33.656852 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:33.656860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:33.657934 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:33.657939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:33.658950 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:33.658966 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:33.659912 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:33.659918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:33.660087 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:33.661131 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:33.661159 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:33.664767 (+ 3608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":293,"threads_started":1}
W20260924 10:08:33.665215 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:33.655357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:55881 (local address 127.6.2.254:34999)
0924 10:08:33.655585 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:33.655593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:33.655886 (+ 293us) server_negotiation.cc:407] Connection header received
0924 10:08:33.656104 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:33.656111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:33.656178 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:33.656284 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:33.656995 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:33.657786 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:33.659130 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:33.659758 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:33.659876 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:33.664973 (+ 5097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:55881: BlockingRecv error: recv got EOF from 127.6.2.194:55881 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 10:08:34 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:34 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:34.154946 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:34.144769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:40841)
0924 10:08:34.145340 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:34.145364 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:34.145602 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:34.146098 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:34.146102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:34.146463 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:34.146644 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:34.146656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:34.148162 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:34.148166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:34.149083 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:34.149095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:34.150295 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:34.150300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:34.150459 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:34.151436 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:34.151456 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:34.154664 (+ 3208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":344,"threads_started":1}
W20260924 10:08:34.155267 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:34.144856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:40841 (local address 127.6.2.254:34999)
0924 10:08:34.145055 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:34.145060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:34.145617 (+ 557us) server_negotiation.cc:407] Connection header received
0924 10:08:34.145736 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:34.145743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:34.145891 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:34.146022 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:34.146850 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:34.147975 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:34.149304 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:34.150059 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:34.150234 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:34.154939 (+ 4705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:40841: BlockingRecv error: recv got EOF from 127.6.2.195:40841 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 10:08:34 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:34 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:34.219363 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:34.210015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:55779)
0924 10:08:34.210559 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:34.210578 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:34.210709 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:34.211225 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:34.211230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:34.211617 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:34.211800 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:34.211810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:34.212864 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:34.212868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:34.213951 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:34.213964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:34.214875 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:34.214880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:34.215020 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:34.215702 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:34.215721 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:34.218936 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":306,"threads_started":1}
W20260924 10:08:34.219496 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:34.210140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:55779 (local address 127.6.2.254:34999)
0924 10:08:34.210339 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:34.210344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:34.210673 (+ 329us) server_negotiation.cc:407] Connection header received
0924 10:08:34.210863 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:34.210868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:34.210939 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:34.211133 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:34.211958 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:34.212727 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:34.214126 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:34.214713 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:34.214823 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:34.219211 (+ 4388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:55779: BlockingRecv error: recv got EOF from 127.6.2.193:55779 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 10:08:34 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:34 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:34.676203 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:34.666323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:47875)
0924 10:08:34.667161 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:34.667179 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:34.667353 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:34.667713 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:34.667717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:34.668145 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:34.668342 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:34.668355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:34.669469 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:34.669476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:34.670581 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:34.670592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:34.671328 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:34.671334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:34.671523 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:34.672123 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:34.672142 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:34.675621 (+ 3479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":453,"threads_started":1}
W20260924 10:08:34.676275 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:34.666542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:47875 (local address 127.6.2.254:34999)
0924 10:08:34.666888 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:34.666895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:34.667276 (+ 381us) server_negotiation.cc:407] Connection header received
0924 10:08:34.667490 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:34.667495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:34.667564 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:34.667688 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:34.668464 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:34.669342 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:34.670717 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:34.671182 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:34.671307 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:34.676034 (+ 4727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:47875: BlockingRecv error: recv got EOF from 127.6.2.194:47875 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
Sep 24 10:08:35 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:35 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:35.164968 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:35.156039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:34405)
0924 10:08:35.156556 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:35.156606 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:35.156774 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:35.157137 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:35.157142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:35.157558 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:35.157715 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:35.157723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:35.158752 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:35.158760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:35.159793 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:35.159804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:35.160525 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:35.160530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:35.160715 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:35.161310 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:35.161328 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:35.164637 (+ 3309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":297,"threads_started":1}
W20260924 10:08:35.165158 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:35.156159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:34405 (local address 127.6.2.254:34999)
0924 10:08:35.156391 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:35.156399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:35.156694 (+ 295us) server_negotiation.cc:407] Connection header received
0924 10:08:35.156897 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:35.156903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:35.156993 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:35.157120 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:35.157855 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:35.158618 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:35.159938 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:35.160375 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:35.160513 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:35.164844 (+ 4331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:34405: BlockingRecv error: recv got EOF from 127.6.2.195:34405 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 10:08:35 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:35 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:35.230273 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:35.220510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:44587)
0924 10:08:35.221120 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:35.221137 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:35.221246 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:35.221852 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:35.221857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:35.222247 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:35.222413 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:35.222422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:35.223545 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:35.223549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:35.224538 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:35.224549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:35.225412 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:35.225421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:35.225643 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:35.226449 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:35.226469 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:35.229911 (+ 3442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":300,"threads_started":1}
W20260924 10:08:35.230347 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:35.220552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:44587 (local address 127.6.2.254:34999)
0924 10:08:35.220741 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:35.220745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:35.221218 (+ 473us) server_negotiation.cc:407] Connection header received
0924 10:08:35.221388 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:35.221393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:35.221453 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:35.221545 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:35.222606 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:35.223405 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:35.224689 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:35.225115 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:35.225225 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:35.230120 (+ 4895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:44587: BlockingRecv error: recv got EOF from 127.6.2.193:44587 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 10:08:35 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:35 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:35.687162 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:35.677315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:38235)
0924 10:08:35.677873 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:35.677890 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:35.677999 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:35.678297 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:35.678300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:35.678692 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:35.678868 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:35.678877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:35.680353 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:35.680357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:35.681317 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:35.681331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:35.682154 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:35.682158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:35.682307 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:35.683036 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:35.683058 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:35.686683 (+ 3625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":297,"threads_started":1}
W20260924 10:08:35.687211 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:35.677575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:38235 (local address 127.6.2.254:34999)
0924 10:08:35.677761 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:35.677766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:35.677994 (+ 228us) server_negotiation.cc:407] Connection header received
0924 10:08:35.678086 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:35.678090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:35.678152 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:35.678238 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:35.679240 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:35.680150 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:35.681520 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:35.681992 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:35.682108 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:35.686904 (+ 4796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:38235: BlockingRecv error: recv got EOF from 127.6.2.194:38235 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 10:08:36 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:36 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:36.179232 15427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:36.166356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:37111)
0924 10:08:36.166968 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:36.166989 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:36.167222 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:36.167702 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:36.167706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:36.168367 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:36.168734 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:36.168747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:36.170535 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:36.170542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:36.172361 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:36.172378 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:36.173401 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:36.173406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:36.173605 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:36.174518 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:36.174544 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:36.178856 (+ 4312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":333,"threads_started":1}
W20260924 10:08:36.179368 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:36.166438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:37111 (local address 127.6.2.254:34999)
0924 10:08:36.166698 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:36.166705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:36.167223 (+ 518us) server_negotiation.cc:407] Connection header received
0924 10:08:36.167398 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:36.167402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:36.167496 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:36.167610 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:36.168948 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:36.170329 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:36.172620 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:36.173182 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:36.173345 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:36.179147 (+ 5802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:37111: BlockingRecv error: recv got EOF from 127.6.2.195:37111 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 10:08:36 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:36 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:36.243001 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:36.231743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:58129)
0924 10:08:36.232519 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:36.232537 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:36.232739 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:36.233226 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:36.233232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:36.233944 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:36.234237 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:36.234251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:36.236019 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:36.236033 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:36.237019 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:36.237042 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:36.237873 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:36.237878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:36.238008 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:36.238719 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:36.238737 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:36.242673 (+ 3936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":451,"threads_started":1}
W20260924 10:08:36.243266 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:36.232002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:58129 (local address 127.6.2.254:34999)
0924 10:08:36.232365 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:36.232373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:36.232647 (+ 274us) server_negotiation.cc:407] Connection header received
0924 10:08:36.232971 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:36.232977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:36.233064 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:36.233181 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:36.234440 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:36.235750 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:36.237207 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:36.237688 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:36.237802 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:36.242976 (+ 5174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:58129: BlockingRecv error: recv got EOF from 127.6.2.193:58129 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Sep 24 10:08:36 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:36 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:36.699597 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:36.688204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:59749)
0924 10:08:36.688766 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:36.688782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:36.688943 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:36.689305 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:36.689309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:36.689726 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:36.689882 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:36.689890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:36.690903 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:36.690910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:36.692587 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:36.692615 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:36.693574 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:36.693579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:36.693779 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:36.694626 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:36.694647 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:36.699171 (+ 4524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":310,"threads_started":1}
W20260924 10:08:36.699818 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:36.688277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:59749 (local address 127.6.2.254:34999)
0924 10:08:36.688566 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:36.688573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:36.688869 (+ 296us) server_negotiation.cc:407] Connection header received
0924 10:08:36.689085 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:36.689091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:36.689164 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:36.689277 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:36.690053 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:36.690767 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:36.692874 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:36.693380 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:36.693541 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:36.699494 (+ 5953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:59749: BlockingRecv error: recv got EOF from 127.6.2.194:59749 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 24 10:08:37 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:37 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:37.197615 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:37.180706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:44047)
0924 10:08:37.181444 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:37.181464 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:37.181619 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:37.182074 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:37.182079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:37.182499 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:37.182735 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:37.182745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:37.184251 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:37.184256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:37.185212 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:37.185225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:37.186038 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:37.186042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:37.186170 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:37.193318 (+ 7148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:37.193341 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:37.197322 (+ 3981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":431,"threads_started":1}
W20260924 10:08:37.197860 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:37.180839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:44047 (local address 127.6.2.254:34999)
0924 10:08:37.181016 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:37.181021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:37.181601 (+ 580us) server_negotiation.cc:407] Connection header received
0924 10:08:37.181759 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:37.181763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:37.181833 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:37.181950 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:37.182905 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:37.184017 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:37.185388 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:37.185845 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:37.185962 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:37.197569 (+ 11607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:44047: BlockingRecv error: recv got EOF from 127.6.2.195:44047 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 10:08:37 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:37 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:37.254096 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:37.244042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:51671)
0924 10:08:37.244716 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:37.244732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:37.244893 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:37.245234 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:37.245238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:37.245710 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:37.245870 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:37.245878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:37.246931 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:37.246937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:37.247859 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:37.247869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:37.248549 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:37.248555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:37.248715 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:37.249361 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:37.249378 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:37.253787 (+ 4409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":292,"threads_started":1}
W20260924 10:08:37.254243 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:37.244158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:51671 (local address 127.6.2.254:34999)
0924 10:08:37.244407 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:37.244415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:37.244823 (+ 408us) server_negotiation.cc:407] Connection header received
0924 10:08:37.245031 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:37.245038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:37.245100 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:37.245205 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:37.246029 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:37.246784 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:37.248017 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:37.248395 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:37.248526 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:37.253989 (+ 5463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:51671: BlockingRecv error: recv got EOF from 127.6.2.193:51671 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 10:08:37 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:37 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:37.710621 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:37.700773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:37493)
0924 10:08:37.701397 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:37.701415 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:37.701539 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:37.702105 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:37.702112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:37.702766 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:37.703126 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:37.703148 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:37.704563 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:37.704571 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:37.705778 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:37.705794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:37.706521 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:37.706525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:37.706683 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:37.707356 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:37.707378 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:37.710354 (+ 2976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":292,"threads_started":1}
W20260924 10:08:37.710856 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:37.700822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:37493 (local address 127.6.2.254:34999)
0924 10:08:37.701062 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:37.701072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:37.701507 (+ 435us) server_negotiation.cc:407] Connection header received
0924 10:08:37.701720 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:37.701724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:37.701809 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:37.701900 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:37.703324 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:37.704384 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:37.705976 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:37.706373 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:37.706478 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:37.710582 (+ 4104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:37493: BlockingRecv error: recv got EOF from 127.6.2.194:37493 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 24 10:08:38 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:38 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:38.207386 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:38.198761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:52837)
0924 10:08:38.199410 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:38.199431 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:38.199544 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:38.199825 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:38.199828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:38.200233 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:38.200396 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:38.200405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:38.201491 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:38.201495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:38.202426 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:38.202436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:38.203324 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:38.203328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:38.203523 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:38.204132 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:38.204149 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:38.207013 (+ 2864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":289,"threads_started":1}
W20260924 10:08:38.207616 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:38.198817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:52837 (local address 127.6.2.254:34999)
0924 10:08:38.199029 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:38.199038 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:38.199526 (+ 488us) server_negotiation.cc:407] Connection header received
0924 10:08:38.199637 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:38.199641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:38.199702 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:38.199786 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:38.200539 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:38.201360 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:38.202571 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:38.203102 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:38.203285 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:38.207381 (+ 4096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:52837: BlockingRecv error: recv got EOF from 127.6.2.195:52837 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 10:08:38 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:38 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:38.264227 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:38.255664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:53647)
0924 10:08:38.256227 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:38.256245 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:38.256368 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:38.256671 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:38.256675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:38.257070 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:38.257212 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:38.257220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:38.258262 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:38.258266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:38.259224 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:38.259234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:38.260081 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:38.260086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:38.260191 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:38.260856 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:38.260874 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:38.263977 (+ 3103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":300,"threads_started":1}
W20260924 10:08:38.264601 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:38.256037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:53647 (local address 127.6.2.254:34999)
0924 10:08:38.256229 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:38.256233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:38.256325 (+ 92us) server_negotiation.cc:407] Connection header received
0924 10:08:38.256465 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:38.256469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:38.256529 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:38.256617 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:38.257359 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:38.258112 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:38.259351 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:38.259889 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:38.259990 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:38.264256 (+ 4266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:53647: BlockingRecv error: recv got EOF from 127.6.2.193:53647 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 10:08:38 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:38 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:38.721916 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:38.712032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:39807)
0924 10:08:38.712860 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:38.712886 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:38.713036 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:38.713617 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:38.713622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:38.714269 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:38.714479 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:38.714490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:38.715889 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:38.715894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:38.716854 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:38.716868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:38.717722 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:38.717727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:38.717854 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:38.718482 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:38.718500 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:38.721557 (+ 3057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":524,"threads_started":1}
W20260924 10:08:38.722048 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:38.712235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:39807 (local address 127.6.2.254:34999)
0924 10:08:38.712505 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:38.712511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:38.713013 (+ 502us) server_negotiation.cc:407] Connection header received
0924 10:08:38.713284 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:38.713289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:38.713364 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:38.713496 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:38.714700 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:38.715658 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:38.717047 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:38.717455 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:38.717579 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:38.721775 (+ 4196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:39807: BlockingRecv error: recv got EOF from 127.6.2.194:39807 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 24 10:08:39 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:39 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:39.219321 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:39.208896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:46605)
0924 10:08:39.209575 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:39.209602 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:39.209817 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:39.210204 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:39.210208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:39.210719 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:39.210900 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:39.210912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:39.212076 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:39.212082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:39.213137 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:39.213149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:39.214098 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:39.214105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:39.214390 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:39.215386 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:39.215411 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:39.219002 (+ 3591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":357,"threads_started":1}
W20260924 10:08:39.219645 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:39.209206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:46605 (local address 127.6.2.254:34999)
0924 10:08:39.209547 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:39.209554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:39.209721 (+ 167us) server_negotiation.cc:407] Connection header received
0924 10:08:39.209970 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:39.209978 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:39.210053 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:39.210174 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:39.211094 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:39.211918 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:39.213299 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:39.213893 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:39.214050 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:39.219320 (+ 5270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:46605: BlockingRecv error: recv got EOF from 127.6.2.195:46605 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 24 10:08:39 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:39 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:39.274171 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:39.265161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:49023)
0924 10:08:39.265786 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:39.265802 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:39.265965 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:39.266325 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:39.266329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:39.266769 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:39.266968 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:39.266976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:39.268042 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:39.268049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:39.268969 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:39.268978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:39.269665 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:39.269671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:39.269848 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:39.270570 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:39.270587 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:39.273925 (+ 3338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":339,"threads_started":1}
W20260924 10:08:39.274415 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:39.265341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:49023 (local address 127.6.2.254:34999)
0924 10:08:39.265618 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:39.265674 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:39.265905 (+ 231us) server_negotiation.cc:407] Connection header received
0924 10:08:39.266119 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:39.266124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:39.266188 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:39.266308 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:39.267160 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:39.267908 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:39.269114 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:39.269509 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:39.269657 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:39.274154 (+ 4497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:49023: BlockingRecv error: recv got EOF from 127.6.2.193:49023 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 24 10:08:39 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:39 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:39.732724 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:39.722857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:36957)
0924 10:08:39.723442 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:39.723458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:39.723571 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:39.723949 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:39.723953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:39.724500 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:39.724820 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:39.724837 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:39.726091 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:39.726095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:39.727118 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:39.727132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:39.727887 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:39.727891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:39.728032 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:39.728815 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:39.728838 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:39.732459 (+ 3621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":327,"threads_started":1}
W20260924 10:08:39.733049 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:39.723004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:36957 (local address 127.6.2.254:34999)
0924 10:08:39.723227 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:39.723238 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:39.723543 (+ 305us) server_negotiation.cc:407] Connection header received
0924 10:08:39.723704 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:39.723708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:39.723768 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:39.723863 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:39.725002 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:39.725963 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:39.727277 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:39.727716 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:39.727821 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:39.732725 (+ 4904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:36957: BlockingRecv error: recv got EOF from 127.6.2.194:36957 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 24 10:08:40 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:40 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:40.230511 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:40.220519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:52359)
0924 10:08:40.221151 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:40.221168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:40.221336 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:40.221703 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:40.221708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:40.222165 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:40.222334 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:40.222343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:40.223424 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:40.223429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:40.224377 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:40.224388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:40.225081 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:40.225088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:40.225274 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:40.225897 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:40.225914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:40.230165 (+ 4251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":343,"threads_started":1}
W20260924 10:08:40.230633 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:40.220620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:52359 (local address 127.6.2.254:34999)
0924 10:08:40.220936 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:40.220945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:40.221261 (+ 316us) server_negotiation.cc:407] Connection header received
0924 10:08:40.221485 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:40.221490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:40.221555 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:40.221665 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:40.222478 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:40.223297 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:40.224527 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:40.224921 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:40.225050 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:40.230371 (+ 5321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:52359: BlockingRecv error: recv got EOF from 127.6.2.195:52359 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Sep 24 10:08:40 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:40 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:40.285142 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:40.275636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:36529)
0924 10:08:40.276508 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:40.276529 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:40.276689 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:40.277224 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:40.277229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:40.277656 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:40.277828 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:40.277838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:40.279247 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:40.279252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:40.280400 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:40.280412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:40.281215 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:40.281220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:40.281340 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:40.281948 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:40.281965 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:40.284784 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":559,"threads_started":1}
W20260924 10:08:40.285256 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:40.275759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:36529 (local address 127.6.2.254:34999)
0924 10:08:40.275943 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:40.275948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:40.276680 (+ 732us) server_negotiation.cc:407] Connection header received
0924 10:08:40.276908 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:40.276913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:40.276990 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:40.277122 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:40.277992 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:40.279027 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:40.280587 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:40.281050 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:40.281161 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:40.285009 (+ 3848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:36529: BlockingRecv error: recv got EOF from 127.6.2.193:36529 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 10:08:40 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:40 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:40.742153 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:40.733754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:35023)
0924 10:08:40.734312 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:40.734329 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:40.734478 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:40.734881 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:40.734889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:40.735397 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:40.735537 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:40.735545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:40.736567 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:40.736571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:40.737527 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:40.737538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:40.738275 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:40.738280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:40.738397 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:40.739029 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:40.739046 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:40.741909 (+ 2863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":310,"threads_started":1}
W20260924 10:08:40.742435 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:40.733875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:35023 (local address 127.6.2.254:34999)
0924 10:08:40.734072 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:40.734081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:40.734410 (+ 329us) server_negotiation.cc:407] Connection header received
0924 10:08:40.734597 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:40.734603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:40.734716 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:40.734820 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:40.735694 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:40.736444 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:40.737701 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:40.738110 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:40.738220 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:40.742100 (+ 3880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:35023: BlockingRecv error: recv got EOF from 127.6.2.194:35023 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 10:08:41 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:41 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:41.241202 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:41.231671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:41771)
0924 10:08:41.232179 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:41.232200 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:41.232327 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:41.232948 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:41.232955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:41.233562 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:41.233741 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:41.233752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:41.234886 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:41.234893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:41.236209 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:41.236232 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:41.237090 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:41.237097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:41.237283 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:41.237971 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:41.237988 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:41.240861 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":269,"threads_started":1}
W20260924 10:08:41.241351 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:41.231874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:41771 (local address 127.6.2.254:34999)
0924 10:08:41.232072 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:41.232077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:41.232381 (+ 304us) server_negotiation.cc:407] Connection header received
0924 10:08:41.232487 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:41.232496 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:41.232635 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:41.232828 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:41.233924 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:41.234720 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:41.236461 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:41.236907 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:41.237065 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:41.241082 (+ 4017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:41771: BlockingRecv error: recv got EOF from 127.6.2.195:41771 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 10:08:41 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:41 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:41.294893 15443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:41.286257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:38185)
0924 10:08:41.286882 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:41.286899 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:41.287089 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:41.287437 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:41.287442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:41.287878 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:41.288046 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:41.288054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:41.289060 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:41.289068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:41.289959 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:41.289969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:41.290744 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:41.290750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:41.290922 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:41.291597 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:41.291615 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:41.294550 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":324,"threads_started":1}
W20260924 10:08:41.295265 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:41.286356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:38185 (local address 127.6.2.254:34999)
0924 10:08:41.286660 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:41.286668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:41.286992 (+ 324us) server_negotiation.cc:407] Connection header received
0924 10:08:41.287237 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:41.287241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:41.287303 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:41.287414 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:41.288192 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:41.288925 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:41.290101 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:41.290538 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:41.290700 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:41.294785 (+ 4085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:38185: BlockingRecv error: recv got EOF from 127.6.2.193:38185 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Sep 24 10:08:41 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:41 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:41.754426 15444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:41.743211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:43669)
0924 10:08:41.743881 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:41.743898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:41.744025 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:41.744630 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:41.744636 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:41.745327 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:41.745528 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:41.745540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:41.746841 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:41.746849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:41.747884 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:41.747896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:41.748617 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:41.748623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:41.748850 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:41.749631 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:41.749666 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:41.754172 (+ 4506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":443,"threads_started":1}
W20260924 10:08:41.754762 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:41.743278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:43669 (local address 127.6.2.254:34999)
0924 10:08:41.743465 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:41.743470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:41.744084 (+ 614us) server_negotiation.cc:407] Connection header received
0924 10:08:41.744199 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:41.744204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:41.744338 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:41.744490 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:41.745756 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:41.746689 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:41.748029 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:41.748452 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:41.748601 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:41.754424 (+ 5823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:43669: BlockingRecv error: recv got EOF from 127.6.2.194:43669 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 10:08:42 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:42 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:42.260191 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:42.242608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:48249)
0924 10:08:42.243401 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:42.243425 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:42.243574 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:42.243938 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:42.243943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:42.244399 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:42.244572 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:42.244582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:42.245592 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:42.245600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:42.246532 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:42.246543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:42.247350 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:42.247356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:42.247543 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:42.256037 (+ 8494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:42.256067 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:42.259822 (+ 3755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":491,"threads_started":1}
W20260924 10:08:42.260277 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:42.242679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:48249 (local address 127.6.2.254:34999)
0924 10:08:42.242921 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:42.242927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:42.243499 (+ 572us) server_negotiation.cc:407] Connection header received
0924 10:08:42.243718 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:42.243723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:42.243797 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:42.243908 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:42.244730 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:42.245456 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:42.246681 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:42.247189 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:42.247329 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:42.260053 (+ 12724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:48249: BlockingRecv error: recv got EOF from 127.6.2.195:48249 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 24 10:08:42 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:42 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:42.305069 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:42.295935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:43503)
0924 10:08:42.296456 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:42.296472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:42.296590 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:42.296996 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:42.296999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:42.297403 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:42.297586 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:42.297595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:42.298652 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:42.298656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:42.299646 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:42.299657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:42.300512 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:42.300517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:42.300657 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:42.301254 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:42.301271 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:42.304721 (+ 3450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":302,"threads_started":1}
W20260924 10:08:42.305241 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:42.296039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:43503 (local address 127.6.2.254:34999)
0924 10:08:42.296345 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:42.296351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:42.296578 (+ 227us) server_negotiation.cc:407] Connection header received
0924 10:08:42.296767 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:42.296771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:42.296834 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:42.296923 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:42.297775 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:42.298498 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:42.299804 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:42.300292 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:42.300394 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:42.304991 (+ 4597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:43503: BlockingRecv error: recv got EOF from 127.6.2.193:43503 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Sep 24 10:08:42 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:42 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:42.766646 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:42.755948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:36909)
0924 10:08:42.756653 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:42.756672 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:42.756896 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:42.757256 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:42.757261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:42.757681 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:42.757844 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:42.757852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:42.758859 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:42.758866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:42.759855 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:42.759867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:42.760762 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:42.760768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:42.761157 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:42.762005 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:42.762072 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:42.766291 (+ 4219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":402,"threads_started":1}
W20260924 10:08:42.766733 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:42.756038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:36909 (local address 127.6.2.254:34999)
0924 10:08:42.756392 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:42.756402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:42.756812 (+ 410us) server_negotiation.cc:407] Connection header received
0924 10:08:42.757036 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:42.757048 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:42.757110 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:42.757231 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:42.757994 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:42.758712 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:42.760000 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:42.760488 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:42.760633 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:42.766497 (+ 5864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:36909: BlockingRecv error: recv got EOF from 127.6.2.194:36909 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 24 10:08:43 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:43 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:43.270148 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:43.261329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:51581)
0924 10:08:43.262068 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:43.262093 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:43.262204 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:43.262502 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:43.262506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:43.262910 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:43.263107 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:43.263115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:43.264136 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:43.264145 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:43.265060 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:43.265070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:43.265775 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:43.265781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:43.265965 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:43.266749 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:43.266767 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:43.269901 (+ 3134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":512,"threads_started":1}
W20260924 10:08:43.270427 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:43.261388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:51581 (local address 127.6.2.254:34999)
0924 10:08:43.261645 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:43.261655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:43.262182 (+ 527us) server_negotiation.cc:407] Connection header received
0924 10:08:43.262302 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:43.262306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:43.262368 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:43.262449 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:43.263247 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:43.263988 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:43.265221 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:43.265625 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:43.265754 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:43.270130 (+ 4376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:51581: BlockingRecv error: recv got EOF from 127.6.2.195:51581 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 10:08:43 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:43 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:43.316578 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:43.306477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:48315)
0924 10:08:43.307140 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:43.307161 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:43.307336 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:43.307681 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:43.307686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:43.308232 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:43.308464 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:43.308478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:43.310153 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:43.310158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:43.311265 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:43.311277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:43.312105 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:43.312110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:43.312277 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:43.312921 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:43.312963 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:43.316348 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":320,"threads_started":1}
W20260924 10:08:43.316823 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:43.306607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:48315 (local address 127.6.2.254:34999)
0924 10:08:43.306856 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:43.306863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:43.307257 (+ 394us) server_negotiation.cc:407] Connection header received
0924 10:08:43.307466 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:43.307472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:43.307542 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:43.307647 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:43.308695 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:43.309950 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:43.311426 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:43.311908 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:43.312066 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:43.316583 (+ 4517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:48315: BlockingRecv error: recv got EOF from 127.6.2.193:48315 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 24 10:08:43 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:43 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:43.777983 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:43.768018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:50975)
0924 10:08:43.768688 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:43.768707 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:43.768886 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:43.769246 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:43.769251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:43.769741 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:43.769910 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:43.769920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:43.771389 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:43.771395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:43.772434 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:43.772445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:43.773327 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:43.773334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:43.773522 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:43.774369 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:43.774389 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:43.777625 (+ 3236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":312,"threads_started":1}
W20260924 10:08:43.778111 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:43.768249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:50975 (local address 127.6.2.254:34999)
0924 10:08:43.768471 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:43.768477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:43.768804 (+ 327us) server_negotiation.cc:407] Connection header received
0924 10:08:43.769018 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:43.769024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:43.769108 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:43.769223 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:43.770074 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:43.771241 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:43.772591 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:43.773115 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:43.773286 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:43.777857 (+ 4571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:50975: BlockingRecv error: recv got EOF from 127.6.2.194:50975 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 24 10:08:44 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:44 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:44.281282 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:44.271358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:59617)
0924 10:08:44.272165 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:44.272189 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:44.272473 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:44.272941 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:44.272945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:44.273537 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:44.273718 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:44.273727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:44.274789 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:44.274793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:44.275821 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:44.275833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:44.277077 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:44.277082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:44.277224 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:44.277876 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:44.277894 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:44.280930 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":440,"threads_started":1}
W20260924 10:08:44.281406 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:44.271554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:59617 (local address 127.6.2.254:34999)
0924 10:08:44.271857 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:44.271865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:44.272414 (+ 549us) server_negotiation.cc:407] Connection header received
0924 10:08:44.272684 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:44.272688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:44.272762 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:44.272902 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:44.273847 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:44.274639 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:44.276102 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:44.276874 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:44.277007 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:44.281160 (+ 4153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:59617: BlockingRecv error: recv got EOF from 127.6.2.195:59617 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 24 10:08:44 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:44 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:44.328653 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:44.317888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:54731)
0924 10:08:44.318545 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:44.318568 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:44.318758 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:44.319139 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:44.319144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:44.319560 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:44.319734 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:44.319742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:44.321391 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:44.321399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:44.322621 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:44.322634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:44.323802 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:44.323817 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:44.323964 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:44.324755 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:44.324771 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:44.328345 (+ 3574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":340,"threads_started":1}
W20260924 10:08:44.329205 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:44.318132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:54731 (local address 127.6.2.254:34999)
0924 10:08:44.318545 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:44.318553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:44.318727 (+ 174us) server_negotiation.cc:407] Connection header received
0924 10:08:44.318918 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:44.318924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:44.318993 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:44.319138 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:44.319883 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:44.320971 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:44.322871 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:44.323613 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:44.323741 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:44.328689 (+ 4948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:54731: BlockingRecv error: recv got EOF from 127.6.2.193:54731 (error 108)
Metrics: {"server-negotiator.queue_time_us":229}
Sep 24 10:08:44 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:44 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:44.788647 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:44.779274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:54497)
0924 10:08:44.779836 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:44.779853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:44.780021 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:44.780367 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:44.780372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:44.780779 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:44.780949 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:44.780958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:44.781982 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:44.781988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:44.782917 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:44.782927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:44.783785 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:44.783790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:44.783966 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:44.784649 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:44.784668 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:44.788049 (+ 3381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":313,"threads_started":1}
W20260924 10:08:44.789242 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:44.779418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:54497 (local address 127.6.2.254:34999)
0924 10:08:44.779710 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:44.779717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:44.779945 (+ 228us) server_negotiation.cc:407] Connection header received
0924 10:08:44.780160 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:44.780166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:44.780227 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:44.780339 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:44.781100 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:44.781838 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:44.783182 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:44.783625 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:44.783774 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:44.788700 (+ 4926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:54497: BlockingRecv error: recv got EOF from 127.6.2.194:54497 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 24 10:08:45 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:45 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.195@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:45.292413 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:45.282435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.195:48625)
0924 10:08:45.282961 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:45.282977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:45.283120 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:45.283521 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:45.283525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:45.283912 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:45.284086 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:45.284094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:45.285258 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:45.285263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:45.286295 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:45.286306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:45.287500 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:45.287507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:45.287687 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:45.288547 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:45.288565 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:45.292131 (+ 3566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":289,"threads_started":1}
W20260924 10:08:45.292941 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:45.282486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:48625 (local address 127.6.2.254:34999)
0924 10:08:45.282677 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:45.282681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:45.283140 (+ 459us) server_negotiation.cc:407] Connection header received
0924 10:08:45.283221 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:45.283224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:45.283308 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:45.283497 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:45.284370 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:45.285090 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:45.286535 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:45.287185 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:45.287341 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:45.292597 (+ 5256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.195:48625: BlockingRecv error: recv got EOF from 127.6.2.195:48625 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 10:08:45 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:45 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.193@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:45.340350 15455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:45.330289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.193:47931)
0924 10:08:45.330850 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:45.330866 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:45.330973 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:45.331582 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:45.331588 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:45.332202 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:45.332504 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:45.332516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:45.333858 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:45.333862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:45.334996 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:45.335009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:45.336083 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:45.336087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:45.336220 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:45.336917 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:45.336935 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:45.340020 (+ 3085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":317,"threads_started":1}
W20260924 10:08:45.340567 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:45.330350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:47931 (local address 127.6.2.254:34999)
0924 10:08:45.330544 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:45.330548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:45.330946 (+ 398us) server_negotiation.cc:407] Connection header received
0924 10:08:45.331228 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:45.331232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:45.331299 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:45.331392 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:45.332889 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:45.333653 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:45.335249 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:45.335856 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:45.336035 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:45.340303 (+ 4268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.193:47931: BlockingRecv error: recv got EOF from 127.6.2.193:47931 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 10:08:45 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:08:45 dist-test-slave-s3n3 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.2.194@KRBTEST.COM for oryx/127.6.2.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:08:45.800395 15456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:45.790548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34999 (local address 127.6.2.194:55717)
0924 10:08:45.791224 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:45.791243 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:45.791375 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:45.792006 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:45.792009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:45.792408 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:45.792626 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:45.792634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:45.793881 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:45.793885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:45.795107 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:45.795129 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:45.796073 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:45.796080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:45.796205 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:45.796982 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:45.797009 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:45.800101 (+ 3092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.2.254:34999: Server oryx/127.6.2.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":366,"threads_started":1}
W20260924 10:08:45.800496 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:45.790672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:55717 (local address 127.6.2.254:34999)
0924 10:08:45.790890 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:45.790894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:45.791417 (+ 523us) server_negotiation.cc:407] Connection header received
0924 10:08:45.791526 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:45.791531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:45.791626 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:45.791966 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:45.792849 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:45.793688 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:45.795262 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:45.795907 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:45.796051 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:45.800298 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.2.194:55717: BlockingRecv error: recv got EOF from 127.6.2.194:55717 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 10:08:46.084780 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 14997
I20260924 10:08:46.098564 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:46.086007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.6.2.193:36629)
0924 10:08:46.086625 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:46.086633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:46.086660 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:08:46.086722 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:46.086729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:46.086967 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:46.087211 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:46.088693 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:46.089697 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:46.090897 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:46.091543 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:46.091685 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:46.094436 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:46.094447 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:46.094462 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:46.094495 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:46.097698 (+ 3203us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.2.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 10:08:46.098207 (+ 509us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59798: Request ticket server kudu/127.6.2.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":287,"threads_started":1}
W20260924 10:08:46.098722 15458 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59798: Request ticket server kudu/127.6.2.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 10:08:46.099014 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 14997: Not authorized: Client connection negotiation failed: client connection to 127.6.2.193:36629: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.2.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 10:08:46.099349 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 14997
I20260924 10:08:46.117210 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 15149
I20260924 10:08:46.134042 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:46.118484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.6.2.194:42231)
0924 10:08:46.119437 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:46.119448 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:46.119495 (+ 47us) server_negotiation.cc:407] Connection header received
0924 10:08:46.119581 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:46.119590 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:46.120043 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:46.120260 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:46.121887 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:46.123006 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:46.124545 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:46.125461 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:46.125638 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:46.129124 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:46.129139 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:46.129160 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:46.129223 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:46.133298 (+ 4075us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.2.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 10:08:46.133722 (+ 424us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43926: Request ticket server kudu/127.6.2.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":414,"threads_started":1}
W20260924 10:08:46.134301 15459 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43926: Request ticket server kudu/127.6.2.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 10:08:46.134311 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 15149: Not authorized: Client connection negotiation failed: client connection to 127.6.2.194:42231: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.2.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 10:08:46.134490 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 15149
I20260924 10:08:46.152731 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 15280
I20260924 10:08:46.167922 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:46.153943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.6.2.195:44199)
0924 10:08:46.154595 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:46.154604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:46.154637 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:08:46.154722 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:46.154729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:46.154982 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:46.155285 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:46.156786 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:46.157937 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:46.159462 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:46.160083 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:46.160235 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:46.163336 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:46.163347 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:46.163364 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:46.163404 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:46.166811 (+ 3407us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.2.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 10:08:46.167495 (+ 684us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56548: Request ticket server kudu/127.6.2.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":316,"threads_started":1}
W20260924 10:08:46.168113 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 15280: Not authorized: Client connection negotiation failed: client connection to 127.6.2.195:44199: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.2.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 10:08:46.168228 15460 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56548: Request ticket server kudu/127.6.2.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 10:08:46.168497 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 15280
I20260924 10:08:46.191152 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 14928
I20260924 10:08:46.193089 14990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:46.317368 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 14928
2026-09-24T10:08:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21241 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T10:08:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:08:46Z Disabled control of system clock
I20260924 10:08:46.411249 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:38167
--webserver_interface=127.6.2.254
--webserver_port=0
--builtin_ntp_servers=127.6.2.212:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:38167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 10:08:46.766644 15465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:46.766988 15465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:46.767109 15465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:46.775048 15465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:46.775202 15465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:46.775240 15465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:46.775292 15465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:46.775338 15465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:08:46.785187 15465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:38167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:38167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:46.786571 15465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:46.788441 15465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:46.803177 15470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:46.803103 15471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:46.803121 15465 server_base.cc:1034] running on GCE node
W20260924 10:08:46.803121 15473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:46.804491 15465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:46.806022 15465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:46.807219 15465 hybrid_clock.cc:648] HybridClock initialized: now 1790244526807148 us; error 63 us; skew 500 ppm
I20260924 10:08:46.807828 15465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:08:46.810447 15465 webserver.cc:460] Webserver started at http://127.6.2.254:40755/ using document root <none> and password file <none>
I20260924 10:08:46.811276 15465 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:46.811372 15465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:46.811714 15465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:46.815533 15465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "a050a378d28a4c1e8476c5ba56736618"
format_stamp: "Formatted at 2026-09-24 10:08:46 on dist-test-slave-s3n3"
server_key: "5e85cfbe44da28f54abd4d87e9052e14"
server_key_iv: "5f4361159698d71598be360c0569c16b"
server_key_version: "encryptionkey@0"
I20260924 10:08:46.816610 15465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "a050a378d28a4c1e8476c5ba56736618"
format_stamp: "Formatted at 2026-09-24 10:08:46 on dist-test-slave-s3n3"
server_key: "5e85cfbe44da28f54abd4d87e9052e14"
server_key_iv: "5f4361159698d71598be360c0569c16b"
server_key_version: "encryptionkey@0"
I20260924 10:08:46.822613 15465 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 10:08:46.826921 15479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:46.829607 15465 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:08:46.829978 15465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "a050a378d28a4c1e8476c5ba56736618"
format_stamp: "Formatted at 2026-09-24 10:08:46 on dist-test-slave-s3n3"
server_key: "5e85cfbe44da28f54abd4d87e9052e14"
server_key_iv: "5f4361159698d71598be360c0569c16b"
server_key_version: "encryptionkey@0"
I20260924 10:08:46.830340 15465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:46.853523 15465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:46.854547 15465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:46.854869 15465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:46.877365 15465 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:38167
I20260924 10:08:46.877473 15530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:38167 every 8 connection(s)
I20260924 10:08:46.878886 15465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:08:46.882964 15531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:46.887557 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 15465
I20260924 10:08:46.887773 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:08:46.888615 6155 external_mini_cluster.cc:1428] Setting key 74afe5946ef002df609767adc32f043e
I20260924 10:08:46.894970 15531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618: Bootstrap starting.
I20260924 10:08:46.898775 15531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:46.900045 15531 log.cc:824] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:46.904247 15531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618: No bootstrap required, opened a new log
I20260924 10:08:46.904841 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:46.891302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.6.2.254:38167)
0924 10:08:46.892368 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:46.892379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:46.892426 (+ 47us) server_negotiation.cc:407] Connection header received
0924 10:08:46.894116 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:46.894140 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:46.894601 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:46.894949 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:46.895696 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:46.896977 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:46.899092 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:46.899827 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:46.899965 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:46.901688 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:46.901705 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:46.901746 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:46.902614 (+ 868us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:46.902727 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:46.902753 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:46.902854 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:46.903454 (+ 600us) server_negotiation.cc:299] Negotiation successful
0924 10:08:46.903794 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":396,"threads_started":1}
I20260924 10:08:46.913336 15531 raft_consensus.cc:348] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a050a378d28a4c1e8476c5ba56736618" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38167 } }
I20260924 10:08:46.913782 15531 raft_consensus.cc:374] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:46.913954 15531 raft_consensus.cc:729] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050a378d28a4c1e8476c5ba56736618, State: Initialized, Role: FOLLOWER
I20260924 10:08:46.914776 15531 consensus_queue.cc:260] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a050a378d28a4c1e8476c5ba56736618" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38167 } }
I20260924 10:08:46.915055 15531 raft_consensus.cc:388] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:08:46.915235 15531 raft_consensus.cc:482] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:08:46.915421 15531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:46.917860 15531 raft_consensus.cc:504] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a050a378d28a4c1e8476c5ba56736618" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38167 } }
I20260924 10:08:46.918469 15531 leader_election.cc:302] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a050a378d28a4c1e8476c5ba56736618; no voters:
I20260924 10:08:46.919338 15531 leader_election.cc:288] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:08:46.919659 15536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:46.920910 15536 raft_consensus.cc:686] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [term 1 LEADER]: Becoming Leader. State: Replica: a050a378d28a4c1e8476c5ba56736618, State: Running, Role: LEADER
I20260924 10:08:46.921603 15536 consensus_queue.cc:237] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a050a378d28a4c1e8476c5ba56736618" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38167 } }
I20260924 10:08:46.921878 15531 sys_catalog.cc:564] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:08:46.925072 15537 sys_catalog.cc:455] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a050a378d28a4c1e8476c5ba56736618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a050a378d28a4c1e8476c5ba56736618" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38167 } } }
I20260924 10:08:46.925536 15537 sys_catalog.cc:458] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:46.925050 15538 sys_catalog.cc:455] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a050a378d28a4c1e8476c5ba56736618. Latest consensus state: current_term: 1 leader_uuid: "a050a378d28a4c1e8476c5ba56736618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a050a378d28a4c1e8476c5ba56736618" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38167 } } }
I20260924 10:08:46.926273 15538 sys_catalog.cc:458] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:46.927855 15545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:08:46.932623 15545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:08:46.942212 15545 catalog_manager.cc:1347] Generated new cluster ID: a232417874c94feabd4fe28834c870a7
I20260924 10:08:46.942351 15545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:08:46.959841 15545 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:08:46.961266 15545 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:08:46.970283 15545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a050a378d28a4c1e8476c5ba56736618: Generated new TSK 0
I20260924 10:08:46.971292 15545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:08:46.985898 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38167
--builtin_ntp_servers=127.6.2.212:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 10:08:47.348078 15555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:47.348381 15555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:47.348464 15555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:47.355911 15555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:47.356026 15555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:47.356129 15555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:08:47.365536 15555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:47.366839 15555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:47.368571 15555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:47.381801 15560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:47.381887 15561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:47.382082 15563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:47.383448 15555 server_base.cc:1034] running on GCE node
I20260924 10:08:47.384196 15555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:47.385221 15555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:47.386521 15555 hybrid_clock.cc:648] HybridClock initialized: now 1790244527386426 us; error 54 us; skew 500 ppm
I20260924 10:08:47.386886 15555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:08:47.397552 15555 webserver.cc:460] Webserver started at http://127.6.2.193:39529/ using document root <none> and password file <none>
I20260924 10:08:47.398439 15555 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:47.398595 15555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:47.398969 15555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:47.402024 15555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "6a6e8045cb44425c845963cdcaecf1fc"
format_stamp: "Formatted at 2026-09-24 10:08:47 on dist-test-slave-s3n3"
server_key: "add00ffed4f238725028422d3b89b893"
server_key_iv: "6acf90eebcdd62c9a209111fb715b4ed"
server_key_version: "encryptionkey@0"
I20260924 10:08:47.402925 15555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "6a6e8045cb44425c845963cdcaecf1fc"
format_stamp: "Formatted at 2026-09-24 10:08:47 on dist-test-slave-s3n3"
server_key: "add00ffed4f238725028422d3b89b893"
server_key_iv: "6acf90eebcdd62c9a209111fb715b4ed"
server_key_version: "encryptionkey@0"
I20260924 10:08:47.409441 15555 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260924 10:08:47.414510 15569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:47.416780 15555 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 10:08:47.417084 15555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "6a6e8045cb44425c845963cdcaecf1fc"
format_stamp: "Formatted at 2026-09-24 10:08:47 on dist-test-slave-s3n3"
server_key: "add00ffed4f238725028422d3b89b893"
server_key_iv: "6acf90eebcdd62c9a209111fb715b4ed"
server_key_version: "encryptionkey@0"
I20260924 10:08:47.417434 15555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:47.436798 15555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:47.438165 15555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:47.438522 15555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:47.440039 15555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:47.441921 15555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:47.442061 15555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:47.442190 15555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:47.442270 15555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:47.487418 15555 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:41135
I20260924 10:08:47.487461 15681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:41135 every 8 connection(s)
I20260924 10:08:47.489442 15555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:08:47.493479 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 15555
I20260924 10:08:47.493705 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:08:47.494633 6155 external_mini_cluster.cc:1428] Setting key 87fa25d4fed812587a02680711a392b9
I20260924 10:08:47.497911 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38167
--builtin_ntp_servers=127.6.2.212:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 10:08:47.508145 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:47.492733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:49361 (local address 127.6.2.254:38167)
0924 10:08:47.493254 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:47.493259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:47.495025 (+ 1766us) server_negotiation.cc:407] Connection header received
0924 10:08:47.497925 (+ 2900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:47.497934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:47.498028 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:47.498162 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:47.499989 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:47.501297 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:47.504405 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:47.504806 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:47.504920 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:47.507039 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:47.507046 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:47.507053 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:47.507119 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:47.507176 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:47.507195 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:47.507337 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:47.507595 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 10:08:47.507713 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":308,"threads_started":1}
I20260924 10:08:47.509344 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:47.493342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:38167 (local address 127.6.2.193:49361)
0924 10:08:47.494778 (+ 1436us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:47.494847 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:47.497544 (+ 2697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:47.498630 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:47.498646 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:47.498787 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:47.499794 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:47.499823 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:47.501639 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:47.501653 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:47.504141 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:47.504165 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:47.505095 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:47.505105 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:47.505457 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:47.506607 (+ 1150us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:08:47.506611 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:47.506676 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:08:47.506778 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:47.507341 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:47.507350 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:47.507357 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:08:47.507541 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 10:08:47.507879 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":454,"threads_started":1}
I20260924 10:08:47.511556 15682 heartbeater.cc:347] Connected to a master server at 127.6.2.254:38167
I20260924 10:08:47.512154 15682 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:47.513461 15682 heartbeater.cc:511] Master 127.6.2.254:38167 requested a full tablet report, sending...
I20260924 10:08:47.516081 15496 ts_manager.cc:194] Registered new tserver with Master: 6a6e8045cb44425c845963cdcaecf1fc (127.6.2.193:41135)
I20260924 10:08:47.517740 15496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.2.193:49361
W20260924 10:08:47.860612 15686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:47.860957 15686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:47.861116 15686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:47.868791 15686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:47.868949 15686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:47.869133 15686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:08:47.878794 15686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:47.880111 15686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:47.881999 15686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:47.897598 15694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:47.897825 15692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:47.898247 15691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:47.899142 15686 server_base.cc:1034] running on GCE node
I20260924 10:08:47.899793 15686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:47.900815 15686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:47.902033 15686 hybrid_clock.cc:648] HybridClock initialized: now 1790244527901966 us; error 57 us; skew 500 ppm
I20260924 10:08:47.902647 15686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:08:47.905851 15686 webserver.cc:460] Webserver started at http://127.6.2.194:35753/ using document root <none> and password file <none>
I20260924 10:08:47.906692 15686 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:47.906791 15686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:47.907151 15686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:47.910048 15686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "5db6b1aed5a9496e9b35889c2c5d7703"
format_stamp: "Formatted at 2026-09-24 10:08:47 on dist-test-slave-s3n3"
server_key: "f536ff3ebc17fda74c058d78c6a93e81"
server_key_iv: "e540f0f2cfd5ed3bb07cc5eaff9538a4"
server_key_version: "encryptionkey@0"
I20260924 10:08:47.910831 15686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "5db6b1aed5a9496e9b35889c2c5d7703"
format_stamp: "Formatted at 2026-09-24 10:08:47 on dist-test-slave-s3n3"
server_key: "f536ff3ebc17fda74c058d78c6a93e81"
server_key_iv: "e540f0f2cfd5ed3bb07cc5eaff9538a4"
server_key_version: "encryptionkey@0"
I20260924 10:08:47.916472 15686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 10:08:47.921020 15700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:47.923432 15686 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260924 10:08:47.923671 15686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "5db6b1aed5a9496e9b35889c2c5d7703"
format_stamp: "Formatted at 2026-09-24 10:08:47 on dist-test-slave-s3n3"
server_key: "f536ff3ebc17fda74c058d78c6a93e81"
server_key_iv: "e540f0f2cfd5ed3bb07cc5eaff9538a4"
server_key_version: "encryptionkey@0"
I20260924 10:08:47.923939 15686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:47.957264 15686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:47.958243 15686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:47.958535 15686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:47.959619 15686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:47.961161 15686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:47.961251 15686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:47.961378 15686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:47.961452 15686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:48.000128 15686 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:44871
I20260924 10:08:48.000177 15812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:44871 every 8 connection(s)
I20260924 10:08:48.001912 15686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:08:48.009347 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 15686
I20260924 10:08:48.009537 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:08:48.010090 6155 external_mini_cluster.cc:1428] Setting key df1cd514963dd78d662fa752ec8314ab
I20260924 10:08:48.014343 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38167
--builtin_ntp_servers=127.6.2.212:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 10:08:48.017032 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:48.004797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:40151 (local address 127.6.2.254:38167)
0924 10:08:48.005026 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:48.005033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:48.006369 (+ 1336us) server_negotiation.cc:407] Connection header received
0924 10:08:48.008171 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:48.008175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:48.008268 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:48.008371 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:48.009898 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.010983 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.013970 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.014391 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.014560 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:48.016229 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:48.016234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:48.016237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:48.016256 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:48.016285 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:48.016296 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:48.016396 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:48.016631 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 10:08:48.016726 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 10:08:48.018334 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:48.005253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:38167 (local address 127.6.2.194:40151)
0924 10:08:48.006206 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:48.006249 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:48.007908 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:48.008592 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:48.008600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:48.008684 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:48.009690 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:48.009716 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.011337 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.011347 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:48.013721 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:48.013753 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.014737 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.014746 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:48.015012 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:48.015865 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:08:48.015867 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:48.015913 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:08:48.016003 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:48.016396 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:48.016409 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:48.016420 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 10:08:48.016626 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 10:08:48.017083 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":317,"threads_started":1}
I20260924 10:08:48.019950 15813 heartbeater.cc:347] Connected to a master server at 127.6.2.254:38167
I20260924 10:08:48.020291 15813 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:48.021047 15813 heartbeater.cc:511] Master 127.6.2.254:38167 requested a full tablet report, sending...
I20260924 10:08:48.022603 15496 ts_manager.cc:194] Registered new tserver with Master: 5db6b1aed5a9496e9b35889c2c5d7703 (127.6.2.194:44871)
I20260924 10:08:48.023756 15496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.2.194:40151
W20260924 10:08:48.362696 15816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:48.362998 15816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:48.363155 15816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:48.370950 15816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:48.371199 15816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:48.371344 15816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:08:48.381207 15816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:48.382579 15816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:48.384416 15816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:48.398073 15821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:48.398171 15822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:48.398116 15824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:48.399376 15816 server_base.cc:1034] running on GCE node
I20260924 10:08:48.400094 15816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:48.401176 15816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:48.402436 15816 hybrid_clock.cc:648] HybridClock initialized: now 1790244528402363 us; error 63 us; skew 500 ppm
I20260924 10:08:48.402822 15816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:08:48.405445 15816 webserver.cc:460] Webserver started at http://127.6.2.195:37863/ using document root <none> and password file <none>
I20260924 10:08:48.406183 15816 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:48.406311 15816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:48.406638 15816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:48.409672 15816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "c4488141a1cd4fc382d5434cccddaa19"
format_stamp: "Formatted at 2026-09-24 10:08:48 on dist-test-slave-s3n3"
server_key: "f646cdb421e67760a221b6a2c55960b8"
server_key_iv: "ad475159ec9ee38fdcad14dc509f8ee6"
server_key_version: "encryptionkey@0"
I20260924 10:08:48.410460 15816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "c4488141a1cd4fc382d5434cccddaa19"
format_stamp: "Formatted at 2026-09-24 10:08:48 on dist-test-slave-s3n3"
server_key: "f646cdb421e67760a221b6a2c55960b8"
server_key_iv: "ad475159ec9ee38fdcad14dc509f8ee6"
server_key_version: "encryptionkey@0"
I20260924 10:08:48.415839 15816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 10:08:48.419473 15830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:48.421060 15816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:08:48.421245 15816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "c4488141a1cd4fc382d5434cccddaa19"
format_stamp: "Formatted at 2026-09-24 10:08:48 on dist-test-slave-s3n3"
server_key: "f646cdb421e67760a221b6a2c55960b8"
server_key_iv: "ad475159ec9ee38fdcad14dc509f8ee6"
server_key_version: "encryptionkey@0"
I20260924 10:08:48.421489 15816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:48.438656 15816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:48.439637 15816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:48.439931 15816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:48.440933 15816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:48.442452 15816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:48.442577 15816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:48.442719 15816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:48.442806 15816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:48.477830 15816 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:41169
I20260924 10:08:48.477876 15942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:41169 every 8 connection(s)
I20260924 10:08:48.479225 15816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:08:48.480783 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 15816
I20260924 10:08:48.480952 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:08:48.481448 6155 external_mini_cluster.cc:1428] Setting key dc6ce79e0bcc5d4a880b9c88ef734a92
I20260924 10:08:48.494469 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:48.482148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:34161 (local address 127.6.2.254:38167)
0924 10:08:48.482372 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:48.482378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:48.484027 (+ 1649us) server_negotiation.cc:407] Connection header received
0924 10:08:48.485918 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:48.485922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:48.485988 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:48.486330 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:48.488128 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.489277 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.491181 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.491768 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.491903 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:48.493556 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:48.493564 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:48.493567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:48.493596 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:48.493646 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:48.493662 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:48.493768 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:48.494126 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 10:08:48.494216 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 10:08:48.495584 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:48.482690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:38167 (local address 127.6.2.195:34161)
0924 10:08:48.483854 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:48.483902 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:48.485646 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:48.486749 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:48.486762 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:48.486896 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:48.487910 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:48.487933 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.489451 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.489455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:48.490977 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:48.490994 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.492076 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.492083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:48.492317 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:48.493015 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:08:48.493018 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:48.493080 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:08:48.493171 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:48.493846 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:48.493854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:48.493861 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:08:48.494026 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 10:08:48.494340 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":400,"threads_started":1}
I20260924 10:08:48.497130 15943 heartbeater.cc:347] Connected to a master server at 127.6.2.254:38167
I20260924 10:08:48.497471 15943 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:48.498272 15943 heartbeater.cc:511] Master 127.6.2.254:38167 requested a full tablet report, sending...
I20260924 10:08:48.500030 15496 ts_manager.cc:194] Registered new tserver with Master: c4488141a1cd4fc382d5434cccddaa19 (127.6.2.195:41169)
I20260924 10:08:48.500913 15496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.2.195:34161
I20260924 10:08:48.510174 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:08:48.517885 15684 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:48.515520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.6.2.254:38167)
0924 10:08:48.515716 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:48.515720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:48.516298 (+ 578us) server_negotiation.cc:407] Connection header received
0924 10:08:48.516413 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:48.516418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:48.516651 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:48.516761 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:48.517478 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35994: BlockingRecv error: recv got EOF from 127.0.0.1:35994 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:08:48.519050 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 15555
I20260924 10:08:48.521171 15682 heartbeater.cc:503] Master 127.6.2.254:38167 was elected leader, sending a full tablet report...
I20260924 10:08:48.529960 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:48.520619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.6.2.193:41135)
0924 10:08:48.521407 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:48.521416 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:48.521451 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:08:48.521519 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:48.521528 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:48.521797 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:48.521972 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:48.522452 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.523930 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.525644 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.526398 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.526525 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:48.527832 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:48.527841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:48.527856 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:48.528613 (+ 757us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:48.528664 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:48.528684 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:48.528790 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:48.529233 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 10:08:48.529547 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":344,"threads_started":1}
I20260924 10:08:48.531703 15677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:48.619390 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 15555
I20260924 10:08:48.637079 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 15686
I20260924 10:08:48.646278 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:48.638171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.6.2.194:44871)
0924 10:08:48.638699 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:48.638706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:48.638731 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:08:48.638795 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:48.638801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:48.639018 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:48.639268 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:48.639656 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.640666 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.641929 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.642706 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.643099 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:48.644662 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:48.644673 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:48.644688 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:48.645339 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:48.645393 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:48.645426 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:48.645563 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:48.645744 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 10:08:48.645895 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":265,"threads_started":1}
I20260924 10:08:48.647603 15808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:48.732155 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 15686
I20260924 10:08:48.751796 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 15816
I20260924 10:08:48.761489 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:48.752789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.6.2.195:41169)
0924 10:08:48.753475 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:48.753490 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:48.753522 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:08:48.753584 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:48.753590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:48.753811 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:48.753960 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:48.754403 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.755570 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.757341 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:48.758271 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:48.758390 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:48.759915 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:48.759922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:48.759937 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:48.760540 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:48.760603 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:48.760624 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:48.760716 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:48.760910 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 10:08:48.761092 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":372,"threads_started":1}
I20260924 10:08:48.762770 15938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:48.847604 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 15816
I20260924 10:08:48.864197 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 15465
I20260924 10:08:48.865622 15526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:48.975492 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 15465
2026-09-24T10:08:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2647 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T10:08:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:08:49Z Disabled control of system clock
I20260924 10:08:49.050565 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:41017
--webserver_interface=127.6.2.254
--webserver_port=0
--builtin_ntp_servers=127.6.2.212:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:41017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 10:08:49.382742 15962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:49.383040 15962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:49.383220 15962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:49.390551 15962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:49.390674 15962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:49.390713 15962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:49.390807 15962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:49.390878 15962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:08:49.400581 15962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:41017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:41017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:49.402063 15962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:49.403951 15962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:49.417045 15968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:49.416996 15967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:49.417523 15970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:49.417899 15962 server_base.cc:1034] running on GCE node
I20260924 10:08:49.418696 15962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:49.420185 15962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:49.421423 15962 hybrid_clock.cc:648] HybridClock initialized: now 1790244529421395 us; error 55 us; skew 500 ppm
I20260924 10:08:49.421790 15962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:08:49.424348 15962 webserver.cc:460] Webserver started at http://127.6.2.254:35997/ using document root <none> and password file <none>
I20260924 10:08:49.425091 15962 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:49.425236 15962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:49.425583 15962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:49.428558 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "ba4b54038d7e4ee5983503436a80eaf0"
format_stamp: "Formatted at 2026-09-24 10:08:49 on dist-test-slave-s3n3"
server_key: "ea98a9d7b8c9d72c2fa8a1f79e94b4e6"
server_key_iv: "0316a8eb60f38ec9e9edaa34906918b2"
server_key_version: "encryptionkey@0"
I20260924 10:08:49.429419 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "ba4b54038d7e4ee5983503436a80eaf0"
format_stamp: "Formatted at 2026-09-24 10:08:49 on dist-test-slave-s3n3"
server_key: "ea98a9d7b8c9d72c2fa8a1f79e94b4e6"
server_key_iv: "0316a8eb60f38ec9e9edaa34906918b2"
server_key_version: "encryptionkey@0"
I20260924 10:08:49.434648 15962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:08:49.438125 15976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:49.439610 15962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:08:49.439797 15962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "ba4b54038d7e4ee5983503436a80eaf0"
format_stamp: "Formatted at 2026-09-24 10:08:49 on dist-test-slave-s3n3"
server_key: "ea98a9d7b8c9d72c2fa8a1f79e94b4e6"
server_key_iv: "0316a8eb60f38ec9e9edaa34906918b2"
server_key_version: "encryptionkey@0"
I20260924 10:08:49.440027 15962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:49.445631 15962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:49.446403 15962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:49.446715 15962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:49.465170 15962 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:41017
I20260924 10:08:49.465225 16027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:41017 every 8 connection(s)
I20260924 10:08:49.466518 15962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:08:49.470220 16028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:49.473645 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 15962
I20260924 10:08:49.473881 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:08:49.474665 6155 external_mini_cluster.cc:1428] Setting key c0b283fd92e3fd0605828bddb4be9ecc
I20260924 10:08:49.483297 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:49.476884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.6.2.254:41017)
0924 10:08:49.477891 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:49.477901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:49.477941 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:08:49.479597 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:49.479620 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:49.480056 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:49.480380 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:49.481519 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:49.481526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:49.481538 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:49.481585 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:49.481644 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:49.481678 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:49.481887 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:49.481983 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 10:08:49.482292 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":367,"threads_started":1}
I20260924 10:08:49.483731 16028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0: Bootstrap starting.
I20260924 10:08:49.486681 16028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:49.487731 16028 log.cc:824] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:49.490429 16028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0: No bootstrap required, opened a new log
I20260924 10:08:49.497625 16028 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b54038d7e4ee5983503436a80eaf0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41017 } }
I20260924 10:08:49.498028 16028 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:49.498143 16028 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4b54038d7e4ee5983503436a80eaf0, State: Initialized, Role: FOLLOWER
I20260924 10:08:49.498772 16028 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b54038d7e4ee5983503436a80eaf0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41017 } }
I20260924 10:08:49.499018 16028 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:08:49.499181 16028 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:08:49.499352 16028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:49.501498 16028 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b54038d7e4ee5983503436a80eaf0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41017 } }
I20260924 10:08:49.502046 16028 leader_election.cc:302] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba4b54038d7e4ee5983503436a80eaf0; no voters:
I20260924 10:08:49.502641 16028 leader_election.cc:288] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:08:49.502812 16033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:49.503649 16033 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [term 1 LEADER]: Becoming Leader. State: Replica: ba4b54038d7e4ee5983503436a80eaf0, State: Running, Role: LEADER
I20260924 10:08:49.504256 16033 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b54038d7e4ee5983503436a80eaf0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41017 } }
I20260924 10:08:49.504789 16028 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:08:49.507010 16035 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba4b54038d7e4ee5983503436a80eaf0. Latest consensus state: current_term: 1 leader_uuid: "ba4b54038d7e4ee5983503436a80eaf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b54038d7e4ee5983503436a80eaf0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41017 } } }
I20260924 10:08:49.507393 16035 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:49.508123 16034 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba4b54038d7e4ee5983503436a80eaf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b54038d7e4ee5983503436a80eaf0" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41017 } } }
I20260924 10:08:49.508400 16040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:08:49.509011 16034 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:49.514788 16040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:08:49.523458 16040 catalog_manager.cc:1347] Generated new cluster ID: a0816ebc823b4686a15ecfe03125f40c
I20260924 10:08:49.523605 16040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:08:49.549715 16040 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:08:49.550257 16040 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:08:49.557327 16040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba4b54038d7e4ee5983503436a80eaf0: Generated new TSK 0
I20260924 10:08:49.558395 16040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:08:49.563706 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41017
--builtin_ntp_servers=127.6.2.212:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 10:08:49.904208 16052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:49.904512 16052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:49.904604 16052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:49.912176 16052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:49.912292 16052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:49.912475 16052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:08:49.922211 16052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:49.923731 16052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:49.925647 16052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:49.939810 16060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:49.939819 16058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:49.939761 16057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:49.940552 16052 server_base.cc:1034] running on GCE node
I20260924 10:08:49.941159 16052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:49.942219 16052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:49.943449 16052 hybrid_clock.cc:648] HybridClock initialized: now 1790244529943404 us; error 43 us; skew 500 ppm
I20260924 10:08:49.943790 16052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:08:49.946376 16052 webserver.cc:460] Webserver started at http://127.6.2.193:35645/ using document root <none> and password file <none>
I20260924 10:08:49.947165 16052 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:49.947288 16052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:49.947645 16052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:49.952334 16052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "c7298c849aa4495a90cc62c49c5f7fd4"
format_stamp: "Formatted at 2026-09-24 10:08:49 on dist-test-slave-s3n3"
server_key: "2128222686154c1cce4b13604314af76"
server_key_iv: "558cd9f739c0c6204b400d6a2a23670c"
server_key_version: "encryptionkey@0"
I20260924 10:08:49.953189 16052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "c7298c849aa4495a90cc62c49c5f7fd4"
format_stamp: "Formatted at 2026-09-24 10:08:49 on dist-test-slave-s3n3"
server_key: "2128222686154c1cce4b13604314af76"
server_key_iv: "558cd9f739c0c6204b400d6a2a23670c"
server_key_version: "encryptionkey@0"
I20260924 10:08:49.958346 16052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:08:49.961862 16066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:49.963493 16052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 10:08:49.963673 16052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "c7298c849aa4495a90cc62c49c5f7fd4"
format_stamp: "Formatted at 2026-09-24 10:08:49 on dist-test-slave-s3n3"
server_key: "2128222686154c1cce4b13604314af76"
server_key_iv: "558cd9f739c0c6204b400d6a2a23670c"
server_key_version: "encryptionkey@0"
I20260924 10:08:49.963915 16052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:49.969817 16052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:49.970633 16052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:49.970911 16052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:49.971858 16052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:49.973276 16052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:49.973400 16052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:49.973541 16052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:49.973639 16052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:50.008606 16052 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:45209
I20260924 10:08:50.008731 16178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:45209 every 8 connection(s)
I20260924 10:08:50.009970 16052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:08:50.018451 16181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:50.012712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:45593 (local address 127.6.2.254:41017)
0924 10:08:50.013161 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:50.013166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:50.014187 (+ 1021us) server_negotiation.cc:407] Connection header received
0924 10:08:50.015973 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:50.015978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:50.016059 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:50.016212 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:50.017570 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:50.017578 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:50.017628 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:50.017673 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:50.017718 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:50.017732 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:50.017879 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:50.018094 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 10:08:50.018159 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":243,"threads_started":1}
I20260924 10:08:50.018877 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 16052
I20260924 10:08:50.019120 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:08:50.019510 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:50.013117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41017 (local address 127.6.2.193:45593)
0924 10:08:50.014016 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:50.014052 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:50.015707 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:50.016434 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:50.016442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:50.016532 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:50.017207 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:08:50.017210 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:50.017246 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:08:50.017335 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:50.017838 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:50.017848 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:50.017856 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:08:50.018151 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 10:08:50.018409 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":270,"threads_started":1}
I20260924 10:08:50.019912 6155 external_mini_cluster.cc:1428] Setting key 0b02080cac3f6636e461394a693e855c
I20260924 10:08:50.021013 16179 heartbeater.cc:347] Connected to a master server at 127.6.2.254:41017
I20260924 10:08:50.021318 16179 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:50.022002 16179 heartbeater.cc:511] Master 127.6.2.254:41017 requested a full tablet report, sending...
I20260924 10:08:50.022742 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41017
--builtin_ntp_servers=127.6.2.212:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 10:08:50.024219 15993 ts_manager.cc:194] Registered new tserver with Master: c7298c849aa4495a90cc62c49c5f7fd4 (127.6.2.193:45209)
W20260924 10:08:50.369451 16183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:50.369747 16183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:50.369901 16183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:50.377508 16183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:50.377631 16183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:50.377813 16183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:08:50.387261 16183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:50.388597 16183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:50.390321 16183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:50.403486 16191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:50.403563 16188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:50.403581 16189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:50.404424 16183 server_base.cc:1034] running on GCE node
I20260924 10:08:50.404996 16183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:50.406262 16183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:50.407498 16183 hybrid_clock.cc:648] HybridClock initialized: now 1790244530407431 us; error 60 us; skew 500 ppm
I20260924 10:08:50.407858 16183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:08:50.410514 16183 webserver.cc:460] Webserver started at http://127.6.2.194:39631/ using document root <none> and password file <none>
I20260924 10:08:50.411346 16183 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:50.411478 16183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:50.411819 16183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:50.414716 16183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "c4e8ca056fed49339d449961af96dea2"
format_stamp: "Formatted at 2026-09-24 10:08:50 on dist-test-slave-s3n3"
server_key: "ab5de9606fd0ca544fcb74c443d0dc40"
server_key_iv: "feb9236be8ed096815834ef2f4c2bd15"
server_key_version: "encryptionkey@0"
I20260924 10:08:50.415616 16183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "c4e8ca056fed49339d449961af96dea2"
format_stamp: "Formatted at 2026-09-24 10:08:50 on dist-test-slave-s3n3"
server_key: "ab5de9606fd0ca544fcb74c443d0dc40"
server_key_iv: "feb9236be8ed096815834ef2f4c2bd15"
server_key_version: "encryptionkey@0"
I20260924 10:08:50.420958 16183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:08:50.424535 16197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:50.426057 16183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:08:50.426236 16183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "c4e8ca056fed49339d449961af96dea2"
format_stamp: "Formatted at 2026-09-24 10:08:50 on dist-test-slave-s3n3"
server_key: "ab5de9606fd0ca544fcb74c443d0dc40"
server_key_iv: "feb9236be8ed096815834ef2f4c2bd15"
server_key_version: "encryptionkey@0"
I20260924 10:08:50.426468 16183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:50.432314 16183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:50.433109 16183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:50.433380 16183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:50.434288 16183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:50.435789 16183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:50.435914 16183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:50.436048 16183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:50.436136 16183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:50.474782 16183 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:38581
I20260924 10:08:50.474820 16309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:38581 every 8 connection(s)
I20260924 10:08:50.476155 16183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:08:50.476830 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 16183
I20260924 10:08:50.477002 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:08:50.477515 6155 external_mini_cluster.cc:1428] Setting key 8177c34a45fae07e65e15eee69faf66a
I20260924 10:08:50.480787 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41017
--builtin_ntp_servers=127.6.2.212:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 10:08:50.488054 16181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:50.480945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:45391 (local address 127.6.2.254:41017)
0924 10:08:50.481246 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:50.481255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:50.482877 (+ 1622us) server_negotiation.cc:407] Connection header received
0924 10:08:50.485712 (+ 2835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:50.485717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:50.485772 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:50.485856 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:50.487406 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:50.487411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:50.487414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:50.487430 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:50.487460 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:50.487471 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:50.487567 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:50.487796 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 10:08:50.487845 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 10:08:50.489374 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:50.480211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41017 (local address 127.6.2.194:45391)
0924 10:08:50.482712 (+ 2501us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:50.482764 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:50.485384 (+ 2620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:50.486144 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:50.486159 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:50.486289 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:50.487002 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:08:50.487008 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:50.487090 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:08:50.487219 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:50.487576 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:50.487588 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:50.487600 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 10:08:50.487873 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 10:08:50.488136 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1854,"thread_start_us":1590,"threads_started":1}
I20260924 10:08:50.490881 16310 heartbeater.cc:347] Connected to a master server at 127.6.2.254:41017
I20260924 10:08:50.491309 16310 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:50.491910 16310 heartbeater.cc:511] Master 127.6.2.254:41017 requested a full tablet report, sending...
I20260924 10:08:50.493491 15993 ts_manager.cc:194] Registered new tserver with Master: c4e8ca056fed49339d449961af96dea2 (127.6.2.194:38581)
W20260924 10:08:50.810344 16313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:50.810676 16313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:50.810837 16313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:50.818410 16313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:50.818565 16313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:50.818755 16313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:08:50.828256 16313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:41017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:50.829644 16313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:50.831619 16313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:50.845134 16319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:50.845180 16318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:50.845108 16321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:50.845920 16313 server_base.cc:1034] running on GCE node
I20260924 10:08:50.846583 16313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:50.847672 16313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:50.848871 16313 hybrid_clock.cc:648] HybridClock initialized: now 1790244530848837 us; error 33 us; skew 500 ppm
I20260924 10:08:50.849191 16313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:08:50.851711 16313 webserver.cc:460] Webserver started at http://127.6.2.195:42979/ using document root <none> and password file <none>
I20260924 10:08:50.852427 16313 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:50.852515 16313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:50.852835 16313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:50.855764 16313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "d746eb8d5949402f860b3e2de5e35e88"
format_stamp: "Formatted at 2026-09-24 10:08:50 on dist-test-slave-s3n3"
server_key: "4a2ecdc939470cc8ead6015774345061"
server_key_iv: "6aead11ad5dfd22e9954267b13982e75"
server_key_version: "encryptionkey@0"
I20260924 10:08:50.856536 16313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "d746eb8d5949402f860b3e2de5e35e88"
format_stamp: "Formatted at 2026-09-24 10:08:50 on dist-test-slave-s3n3"
server_key: "4a2ecdc939470cc8ead6015774345061"
server_key_iv: "6aead11ad5dfd22e9954267b13982e75"
server_key_version: "encryptionkey@0"
I20260924 10:08:50.861680 16313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:08:50.865250 16327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:50.866951 16313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:08:50.867167 16313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "d746eb8d5949402f860b3e2de5e35e88"
format_stamp: "Formatted at 2026-09-24 10:08:50 on dist-test-slave-s3n3"
server_key: "4a2ecdc939470cc8ead6015774345061"
server_key_iv: "6aead11ad5dfd22e9954267b13982e75"
server_key_version: "encryptionkey@0"
I20260924 10:08:50.867362 16313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:50.873155 16313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:50.873977 16313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:50.874251 16313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:50.875224 16313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:50.876667 16313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:50.876751 16313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:50.876897 16313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:50.876989 16313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:50.911860 16313 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:43783
I20260924 10:08:50.911912 16439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:43783 every 8 connection(s)
I20260924 10:08:50.913218 16313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:08:50.917460 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 16313
I20260924 10:08:50.917681 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:08:50.918138 6155 external_mini_cluster.cc:1428] Setting key 6004e7e3136d26e2c0fc2b7d5e1e7a4b
I20260924 10:08:50.922115 16181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:50.915823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:36349 (local address 127.6.2.254:41017)
0924 10:08:50.916003 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:50.916007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:50.917367 (+ 1360us) server_negotiation.cc:407] Connection header received
0924 10:08:50.919637 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:50.919642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:50.919707 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:50.919836 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:50.921383 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:50.921389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:50.921395 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:50.921415 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:50.921443 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:50.921454 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:50.921580 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:50.921787 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 10:08:50.921851 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:08:50.923136 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:50.916312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:41017 (local address 127.6.2.195:36349)
0924 10:08:50.917202 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:50.917244 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:50.919321 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:50.920095 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:50.920111 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:50.920261 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:50.921068 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:08:50.921071 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:50.921120 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:08:50.921220 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:50.921555 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:50.921565 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:50.921573 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:08:50.921877 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 10:08:50.922125 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":272,"threads_started":1}
I20260924 10:08:50.924481 16440 heartbeater.cc:347] Connected to a master server at 127.6.2.254:41017
I20260924 10:08:50.924814 16440 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:50.925348 16440 heartbeater.cc:511] Master 127.6.2.254:41017 requested a full tablet report, sending...
I20260924 10:08:50.926857 15993 ts_manager.cc:194] Registered new tserver with Master: d746eb8d5949402f860b3e2de5e35e88 (127.6.2.195:43783)
I20260924 10:08:50.933399 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:08:50.939502 16181 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:08:50.937831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47970 (local address 127.6.2.254:41017)
0924 10:08:50.938014 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:50.938018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:50.938569 (+ 551us) server_negotiation.cc:407] Connection header received
0924 10:08:50.938702 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:50.938706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:50.938752 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:50.938835 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:50.939218 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47970: BlockingRecv error: recv got EOF from 127.0.0.1:47970 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:08:50.940858 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 16052
I20260924 10:08:50.944351 16449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:50.941725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.6.2.193:45209)
0924 10:08:50.942242 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:50.942250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:50.942275 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:08:50.942331 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:50.942337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:50.942543 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:50.942676 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:50.943663 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:50.943688 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:50.943700 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:50.943719 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:50.943760 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:50.943775 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:50.943879 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:50.943987 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 10:08:50.944062 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":265,"threads_started":1}
I20260924 10:08:50.945490 16174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:51.027271 16179 heartbeater.cc:503] Master 127.6.2.254:41017 was elected leader, sending a full tablet report...
I20260924 10:08:51.027568 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 16052
W20260924 10:08:51.045563 15978 connection.cc:535] server connection from 127.6.2.193:45593 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:08:51.045903 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 16183
I20260924 10:08:51.049360 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:51.046832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.6.2.194:38581)
0924 10:08:51.047374 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:51.047381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:51.047407 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:08:51.047467 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:51.047473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:51.047694 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:51.047819 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:51.048721 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:51.048728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:51.048740 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:51.048757 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:51.048797 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:51.048811 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:51.048906 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:51.049044 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 10:08:51.049122 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":301,"threads_started":1}
I20260924 10:08:51.050550 16305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:51.131315 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 16183
I20260924 10:08:51.147372 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 16313
I20260924 10:08:51.151305 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:51.148356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.6.2.195:43783)
0924 10:08:51.149097 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:51.149104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:51.149136 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:08:51.149196 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:51.149202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:51.149415 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:51.149548 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:51.150489 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:51.150496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:51.150508 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:08:51.150520 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:51.150576 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:08:51.150592 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:51.150724 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:51.150799 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 10:08:51.150915 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":423,"threads_started":1}
I20260924 10:08:51.152598 16435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:51.237648 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 16313
I20260924 10:08:51.254134 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 15962
I20260924 10:08:51.255434 16023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:51.357170 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 15962
2026-09-24T10:08:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2379 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:08:51 dist-test-slave-s3n3 krb5kdc[16458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:08:51 dist-test-slave-s3n3 krb5kdc[16458](info): set up 2 sockets
Sep 24 10:08:51 dist-test-slave-s3n3 krb5kdc[16458](info): commencing operation
krb5kdc: starting...
W20260924 10:08:52.488582 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:08:52 dist-test-slave-s3n3 krb5kdc[16458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:08:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:08:52.738361 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:33887
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:39493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:33887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:53.066886 16471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:53.067273 16471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:53.067451 16471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:53.074608 16471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:08:53.074779 16471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:53.074893 16471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:53.075002 16471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:08:53.075109 16471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:08:53.085289 16471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:39493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:33887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:33887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:53.086637 16471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:53.087950 16471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:53.100003 16477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:53.100041 16479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:53.100826 16476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:53.109246 16471 server_base.cc:1034] running on GCE node
I20260924 10:08:53.110210 16471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:53.111882 16471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:53.113159 16471 hybrid_clock.cc:648] HybridClock initialized: now 1790244533113093 us; error 61 us; skew 500 ppm
I20260924 10:08:53.113520 16471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:53 dist-test-slave-s3n3 krb5kdc[16458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244533, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:53.125962 16471 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:08:53.127803 16471 webserver.cc:460] Webserver started at http://127.6.2.254:36149/ using document root <none> and password file <none>
I20260924 10:08:53.128557 16471 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:53.128716 16471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:53.129025 16471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:53.132261 16471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "e0f27b7095884b18b3ecdae0911011c6"
format_stamp: "Formatted at 2026-09-24 10:08:53 on dist-test-slave-s3n3"
server_key: "49e532c59d6eed4edbe116297e30a29c"
server_key_iv: "f102ac895a89a12be55acadab5e7b812"
server_key_version: "encryptionkey@0"
I20260924 10:08:53.133028 16471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "e0f27b7095884b18b3ecdae0911011c6"
format_stamp: "Formatted at 2026-09-24 10:08:53 on dist-test-slave-s3n3"
server_key: "49e532c59d6eed4edbe116297e30a29c"
server_key_iv: "f102ac895a89a12be55acadab5e7b812"
server_key_version: "encryptionkey@0"
I20260924 10:08:53.138531 16471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:08:53.142207 16486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:53.143743 16471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:08:53.143899 16471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "e0f27b7095884b18b3ecdae0911011c6"
format_stamp: "Formatted at 2026-09-24 10:08:53 on dist-test-slave-s3n3"
server_key: "49e532c59d6eed4edbe116297e30a29c"
server_key_iv: "f102ac895a89a12be55acadab5e7b812"
server_key_version: "encryptionkey@0"
I20260924 10:08:53.144069 16471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:53.171622 16471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:53.176329 16471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:53.176577 16471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:53.195430 16471 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:33887
I20260924 10:08:53.195469 16537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:33887 every 8 connection(s)
I20260924 10:08:53.196857 16471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:08:53.200663 16538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:53.202023 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 16471
I20260924 10:08:53.202277 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:08:53.203118 6155 external_mini_cluster.cc:1428] Setting key 63cf18efb744c764f1cb3c03541a88b6
I20260924 10:08:53.215646 16538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6: Bootstrap starting.
Sep 24 10:08:53 dist-test-slave-s3n3 krb5kdc[16458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:08:53.219890 16538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:53.221424 16538 log.cc:824] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:53.224714 16538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6: No bootstrap required, opened a new log
I20260924 10:08:53.228130 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:53.205187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.6.2.254:33887)
0924 10:08:53.206101 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:53.206115 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:53.206157 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:08:53.207754 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:53.207785 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:53.208179 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:53.208499 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:53.210313 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:53.211703 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:53.213566 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:53.214239 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:53.214361 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:53.219581 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:53.219596 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:53.219618 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:53.219671 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:53.223752 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:53.224651 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:53.224658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:53.224707 (+ 49us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:53.224961 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:53.225394 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:53.225398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:53.225400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:53.225933 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:53.226173 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:53.226627 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 10:08:53.227007 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":497,"threads_started":1}
I20260924 10:08:53.233907 16538 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f27b7095884b18b3ecdae0911011c6" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33887 } }
I20260924 10:08:53.234295 16538 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:53.234400 16538 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f27b7095884b18b3ecdae0911011c6, State: Initialized, Role: FOLLOWER
I20260924 10:08:53.234998 16538 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f27b7095884b18b3ecdae0911011c6" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33887 } }
I20260924 10:08:53.235294 16538 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:08:53.235441 16538 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:08:53.235625 16538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:53.237819 16538 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f27b7095884b18b3ecdae0911011c6" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33887 } }
I20260924 10:08:53.238335 16538 leader_election.cc:302] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0f27b7095884b18b3ecdae0911011c6; no voters:
I20260924 10:08:53.239001 16538 leader_election.cc:288] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:08:53.239207 16543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:53.239946 16543 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [term 1 LEADER]: Becoming Leader. State: Replica: e0f27b7095884b18b3ecdae0911011c6, State: Running, Role: LEADER
I20260924 10:08:53.240527 16543 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f27b7095884b18b3ecdae0911011c6" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33887 } }
I20260924 10:08:53.241324 16538 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:08:53.243888 16545 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f27b7095884b18b3ecdae0911011c6. Latest consensus state: current_term: 1 leader_uuid: "e0f27b7095884b18b3ecdae0911011c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f27b7095884b18b3ecdae0911011c6" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33887 } } }
I20260924 10:08:53.244051 16544 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0f27b7095884b18b3ecdae0911011c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f27b7095884b18b3ecdae0911011c6" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33887 } } }
I20260924 10:08:53.244505 16545 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:53.244535 16544 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:08:53.248332 16551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:08:53.253515 16551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:08:53.262943 16551 catalog_manager.cc:1347] Generated new cluster ID: 09078cb7909e40abacf2c427d98db94d
I20260924 10:08:53.263098 16551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:08:53.279831 16551 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:08:53.281075 16551 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:08:53.290004 16551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6: Generated new TSK 0
I20260924 10:08:53.290838 16551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:08:53.362895 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33887
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:39493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:53.719278 16564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:53.719573 16564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:53.719694 16564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:53.727203 16564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:53.727322 16564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:53.727494 16564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:08:53.737947 16564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:39493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:53.739482 16564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:53.740833 16564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:53.754776 16569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:53.754904 16570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:53.755246 16572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:53.756762 16564 server_base.cc:1034] running on GCE node
I20260924 10:08:53.758160 16564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:53.759840 16564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:53.761096 16564 hybrid_clock.cc:648] HybridClock initialized: now 1790244533761057 us; error 59 us; skew 500 ppm
I20260924 10:08:53.761493 16564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:53 dist-test-slave-s3n3 krb5kdc[16458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244533, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:53.765033 16564 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:08:53.766697 16564 webserver.cc:460] Webserver started at http://127.6.2.193:37893/ using document root <none> and password file <none>
I20260924 10:08:53.767529 16564 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:53.767699 16564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:53.768013 16564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:53.771155 16564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "e51167721a9b4b09a6614a3bf622e8b1"
format_stamp: "Formatted at 2026-09-24 10:08:53 on dist-test-slave-s3n3"
server_key: "fb04a6abae33165dfe89ca1cda3e1837"
server_key_iv: "6d1e9680e42512ac169864c4ab2074fa"
server_key_version: "encryptionkey@0"
I20260924 10:08:53.772392 16564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "e51167721a9b4b09a6614a3bf622e8b1"
format_stamp: "Formatted at 2026-09-24 10:08:53 on dist-test-slave-s3n3"
server_key: "fb04a6abae33165dfe89ca1cda3e1837"
server_key_iv: "6d1e9680e42512ac169864c4ab2074fa"
server_key_version: "encryptionkey@0"
I20260924 10:08:53.777984 16564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:08:53.781694 16579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:53.783340 16564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:08:53.783527 16564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "e51167721a9b4b09a6614a3bf622e8b1"
format_stamp: "Formatted at 2026-09-24 10:08:53 on dist-test-slave-s3n3"
server_key: "fb04a6abae33165dfe89ca1cda3e1837"
server_key_iv: "6d1e9680e42512ac169864c4ab2074fa"
server_key_version: "encryptionkey@0"
I20260924 10:08:53.783784 16564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:53.803403 16564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:53.807931 16564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:53.808245 16564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:53.809238 16564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:53.810570 16564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:53.810675 16564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:53.810830 16564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:53.810904 16564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:53.847651 16564 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:40273
I20260924 10:08:53.847695 16691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:40273 every 8 connection(s)
I20260924 10:08:53.849190 16564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:08:53.850680 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 16564
I20260924 10:08:53.850906 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:08:53.851708 6155 external_mini_cluster.cc:1428] Setting key d12e8c8184193c77d4a3e036f014321d
Sep 24 10:08:53 dist-test-slave-s3n3 krb5kdc[16458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244533, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:08:53.871309 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:53.851913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:53741 (local address 127.6.2.254:33887)
0924 10:08:53.852613 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:53.852621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:53.853591 (+ 970us) server_negotiation.cc:407] Connection header received
0924 10:08:53.855414 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:53.855422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:53.855513 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:53.855650 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:53.857201 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:53.858351 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:53.860154 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:53.860649 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:53.860753 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:53.865068 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:53.865074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:53.865079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:53.865119 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:53.867902 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:53.868876 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:53.868880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:53.868882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:53.868995 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:53.869469 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:53.869472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:53.869474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:53.869724 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:53.869829 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:53.870804 (+ 975us) server_negotiation.cc:299] Negotiation successful
0924 10:08:53.870975 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":444,"threads_started":1}
I20260924 10:08:53.872867 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:53.852616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:33887 (local address 127.6.2.193:53741)
0924 10:08:53.853414 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:53.853469 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:53.855146 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:53.855885 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:53.855893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:53.856423 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:53.857007 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:53.857027 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:53.858582 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:53.858586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:53.859978 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:53.859991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:53.860904 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:53.860914 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:53.861236 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:53.862096 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:53.862126 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:53.864765 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:53.868058 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:53.868070 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:53.868092 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:53.868744 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:53.869103 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:53.869110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:53.869116 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:53.869355 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:53.869853 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:53.869864 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:53.870356 (+ 492us) client_negotiation.cc:769] Sending connection context
0924 10:08:53.870687 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 10:08:53.871336 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":338,"threads_started":1}
I20260924 10:08:53.874568 16692 heartbeater.cc:347] Connected to a master server at 127.6.2.254:33887
I20260924 10:08:53.874871 16692 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:53.875644 16692 heartbeater.cc:511] Master 127.6.2.254:33887 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
I20260924 10:08:53.877833 16503 ts_manager.cc:194] Registered new tserver with Master: e51167721a9b4b09a6614a3bf622e8b1 (127.6.2.193:40273)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
I20260924 10:08:53.879580 16503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:53741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:08:53.907562 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33887
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:39493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:54.234128 16698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:54.234494 16698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:54.234656 16698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:54.242203 16698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:54.242309 16698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:54.242512 16698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:08:54.252616 16698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:39493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:54.253865 16698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:54.255141 16698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:54.268121 16704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:54.268146 16703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:54.268138 16706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:54.269207 16698 server_base.cc:1034] running on GCE node
I20260924 10:08:54.269797 16698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:54.270807 16698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:54.272032 16698 hybrid_clock.cc:648] HybridClock initialized: now 1790244534271965 us; error 56 us; skew 500 ppm
I20260924 10:08:54.272439 16698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:54 dist-test-slave-s3n3 krb5kdc[16458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244534, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:54.276769 16698 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:08:54.278767 16698 webserver.cc:460] Webserver started at http://127.6.2.194:40147/ using document root <none> and password file <none>
I20260924 10:08:54.279820 16698 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:54.279968 16698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:54.280244 16698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:54.283421 16698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "4cc2f88ef8a14542a69b776e1194b040"
format_stamp: "Formatted at 2026-09-24 10:08:54 on dist-test-slave-s3n3"
server_key: "88c70795576715831c6143cd36e1a17d"
server_key_iv: "4c974d11e0f15c6943b1e90defa77610"
server_key_version: "encryptionkey@0"
I20260924 10:08:54.284205 16698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "4cc2f88ef8a14542a69b776e1194b040"
format_stamp: "Formatted at 2026-09-24 10:08:54 on dist-test-slave-s3n3"
server_key: "88c70795576715831c6143cd36e1a17d"
server_key_iv: "4c974d11e0f15c6943b1e90defa77610"
server_key_version: "encryptionkey@0"
I20260924 10:08:54.290031 16698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:08:54.293666 16713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:54.295042 16698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:08:54.295288 16698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "4cc2f88ef8a14542a69b776e1194b040"
format_stamp: "Formatted at 2026-09-24 10:08:54 on dist-test-slave-s3n3"
server_key: "88c70795576715831c6143cd36e1a17d"
server_key_iv: "4c974d11e0f15c6943b1e90defa77610"
server_key_version: "encryptionkey@0"
I20260924 10:08:54.295488 16698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:54.306427 16698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:54.310199 16698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:54.310480 16698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:54.311440 16698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:54.312794 16698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:54.312896 16698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:54.313050 16698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:54.313122 16698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:54.346467 16698 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:42667
I20260924 10:08:54.346509 16825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:42667 every 8 connection(s)
I20260924 10:08:54.347800 16698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:08:54.350282 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 16698
I20260924 10:08:54.350479 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:08:54.351032 6155 external_mini_cluster.cc:1428] Setting key a2ed2dbf7d4d3fa9364b69e71ccb8b57
Sep 24 10:08:54 dist-test-slave-s3n3 krb5kdc[16458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244534, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:08:54.368999 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:54.351322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:52455 (local address 127.6.2.254:33887)
0924 10:08:54.351501 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:54.351505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:54.352459 (+ 954us) server_negotiation.cc:407] Connection header received
0924 10:08:54.354479 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:54.354484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:54.354575 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:54.354685 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:54.356236 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.356941 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.358537 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.358951 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.359115 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:54.363198 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:54.363202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:54.363206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:54.363237 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:54.365832 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:54.366584 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:54.366596 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:54.366599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:54.366725 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:54.367239 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:54.367244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:54.367246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:54.367559 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:54.367715 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:54.368422 (+ 707us) server_negotiation.cc:299] Negotiation successful
0924 10:08:54.368650 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:08:54.370036 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:54.351434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:33887 (local address 127.6.2.194:52455)
0924 10:08:54.352285 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:54.352332 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:54.354215 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:54.354918 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:54.354926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:54.355493 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:54.356069 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:54.356087 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.357104 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.357109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:54.358375 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:54.358390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.359227 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.359235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:54.359508 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:54.360324 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:54.360357 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:54.362928 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:54.365991 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:54.366001 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:54.366021 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:54.366450 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:54.366847 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:54.366852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:54.366856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:54.367045 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:54.367690 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:54.367699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:54.368053 (+ 354us) client_negotiation.cc:769] Sending connection context
0924 10:08:54.368321 (+ 268us) client_negotiation.cc:240] Negotiation successful
0924 10:08:54.368887 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":338,"threads_started":1}
I20260924 10:08:54.371616 16826 heartbeater.cc:347] Connected to a master server at 127.6.2.254:33887
I20260924 10:08:54.371987 16826 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:54.372944 16826 heartbeater.cc:511] Master 127.6.2.254:33887 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:08:54.374612 16503 ts_manager.cc:194] Registered new tserver with Master: 4cc2f88ef8a14542a69b776e1194b040 (127.6.2.194:42667)
I20260924 10:08:54.375557 16503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:52455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:08:54.407279 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33887
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:39493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:08:54.732950 16831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:08:54.733233 16831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:08:54.733392 16831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:08:54.740885 16831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:08:54.740988 16831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:08:54.741199 16831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:08:54.751436 16831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:39493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:08:54.752692 16831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:08:54.753943 16831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:08:54.767160 16837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:54.767314 16836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:08:54.767679 16839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:08:54.772794 16831 server_base.cc:1034] running on GCE node
I20260924 10:08:54.773424 16831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:08:54.774420 16831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:08:54.775636 16831 hybrid_clock.cc:648] HybridClock initialized: now 1790244534775555 us; error 51 us; skew 500 ppm
I20260924 10:08:54.775954 16831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:08:54 dist-test-slave-s3n3 krb5kdc[16458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244534, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:08:54.779170 16831 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:08:54.780681 16831 webserver.cc:460] Webserver started at http://127.6.2.195:38597/ using document root <none> and password file <none>
I20260924 10:08:54.781390 16831 fs_manager.cc:359] Metadata directory not provided
I20260924 10:08:54.781509 16831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:08:54.781804 16831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:08:54.784788 16831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "a8f870d3b5ca452fbf6e4168092e1302"
format_stamp: "Formatted at 2026-09-24 10:08:54 on dist-test-slave-s3n3"
server_key: "54eb18618540005112b24791b20375c3"
server_key_iv: "ec114a869d6ed7f804f1c7bf729aa626"
server_key_version: "encryptionkey@0"
I20260924 10:08:54.785552 16831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "a8f870d3b5ca452fbf6e4168092e1302"
format_stamp: "Formatted at 2026-09-24 10:08:54 on dist-test-slave-s3n3"
server_key: "54eb18618540005112b24791b20375c3"
server_key_iv: "ec114a869d6ed7f804f1c7bf729aa626"
server_key_version: "encryptionkey@0"
I20260924 10:08:54.790679 16831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:08:54.794097 16846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:54.795442 16831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 10:08:54.795593 16831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "a8f870d3b5ca452fbf6e4168092e1302"
format_stamp: "Formatted at 2026-09-24 10:08:54 on dist-test-slave-s3n3"
server_key: "54eb18618540005112b24791b20375c3"
server_key_iv: "ec114a869d6ed7f804f1c7bf729aa626"
server_key_version: "encryptionkey@0"
I20260924 10:08:54.795773 16831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:08:54.808645 16831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:08:54.812848 16831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:08:54.813148 16831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:08:54.814092 16831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:08:54.815452 16831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:08:54.815522 16831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:54.815692 16831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:08:54.815753 16831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:08:54.850994 16831 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:36173
I20260924 10:08:54.851016 16958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:36173 every 8 connection(s)
I20260924 10:08:54.852352 16831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:08:54.852672 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 16831
I20260924 10:08:54.852867 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:08:54.853349 6155 external_mini_cluster.cc:1428] Setting key 7ec1324baf6a2a7b38986dbb98295fe9
Sep 24 10:08:54 dist-test-slave-s3n3 krb5kdc[16458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244534, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:08:54.874527 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:54.856938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:42971 (local address 127.6.2.254:33887)
0924 10:08:54.857128 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:54.857132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:54.858271 (+ 1139us) server_negotiation.cc:407] Connection header received
0924 10:08:54.860177 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:54.860182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:54.860303 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:54.860415 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:54.861892 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.862651 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.864416 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.864835 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.864976 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:54.868732 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:54.868737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:54.868740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:54.868771 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:54.871775 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:54.872495 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:54.872499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:54.872501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:54.872624 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:54.873022 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:54.873025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:54.873027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:54.873299 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:54.873404 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:54.874118 (+ 714us) server_negotiation.cc:299] Negotiation successful
0924 10:08:54.874290 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:08:54.875605 16961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:54.857407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:33887 (local address 127.6.2.195:42971)
0924 10:08:54.858120 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:54.858162 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:54.859908 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:54.860654 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:54.860662 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:54.861164 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:08:54.861728 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:54.861746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.862813 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.862817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:54.864262 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:54.864275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.865070 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.865077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:54.865320 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:54.865986 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:08:54.866009 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:08:54.868503 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:08:54.871913 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:54.871920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:54.871934 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:54.872365 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:54.872728 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:08:54.872731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:08:54.872733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:08:54.872922 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:08:54.873409 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:08:54.873414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:08:54.873786 (+ 372us) client_negotiation.cc:769] Sending connection context
0924 10:08:54.874008 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 10:08:54.874398 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":280,"threads_started":1}
I20260924 10:08:54.877177 16959 heartbeater.cc:347] Connected to a master server at 127.6.2.254:33887
I20260924 10:08:54.877486 16959 heartbeater.cc:464] Registering TS with master...
I20260924 10:08:54.878284 16959 heartbeater.cc:511] Master 127.6.2.254:33887 requested a full tablet report, sending...
I20260924 10:08:54.879973 16503 ts_manager.cc:194] Registered new tserver with Master: a8f870d3b5ca452fbf6e4168092e1302 (127.6.2.195:36173)
I20260924 10:08:54.881021 16503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:42971
I20260924 10:08:54.882210 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:08:54.882503 16692 heartbeater.cc:503] Master 127.6.2.254:33887 was elected leader, sending a full tablet report...
I20260924 10:08:54.902072 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:54.887444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.6.2.254:33887)
0924 10:08:54.887661 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:54.887671 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:54.888116 (+ 445us) server_negotiation.cc:407] Connection header received
0924 10:08:54.888270 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:54.888274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:54.888332 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:54.888410 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:54.889798 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.890504 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.891786 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.892188 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.892282 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:54.894477 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:54.894481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:54.894485 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:54.894515 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:54.897874 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:54.899393 (+ 1519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:54.899399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:54.899402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:54.899663 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:54.900446 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:54.900453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:54.900457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:54.900830 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:54.901146 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:54.901483 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 10:08:54.901716 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:08:54.908893 16502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:08:54.916399 16502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:08:54.941336 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:54.931699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.6.2.193:40273)
0924 10:08:54.933006 (+ 1307us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:54.933013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:54.933853 (+ 840us) server_negotiation.cc:407] Connection header received
0924 10:08:54.933957 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:54.933965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:54.934176 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:54.934344 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:54.935208 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.936734 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.939107 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.940030 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.940207 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:54.940446 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:54.940825 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 10:08:54.940992 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":1018,"threads_started":1}
I20260924 10:08:54.941382 16971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:54.930281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:42667 (local address 127.0.0.1:41248)
0924 10:08:54.931726 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:54.931770 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:54.931967 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:54.933269 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:54.933276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:54.933314 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:54.933857 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:54.933872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.936049 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.936059 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:54.938347 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:54.938362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.940333 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.940339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:54.940536 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:54.940584 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 10:08:54.940657 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 10:08:54.940859 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"mutex_wait_us":5,"thread_start_us":702,"threads_started":1}
I20260924 10:08:54.941883 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:54.930876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.6.2.194:42667)
0924 10:08:54.932594 (+ 1718us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:54.932606 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:54.932637 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:08:54.932745 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:54.932756 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:54.933013 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:54.934115 (+ 1102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:54.934344 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.935923 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.938528 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.940096 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.941127 (+ 1031us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:54.941299 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:54.941358 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 10:08:54.941529 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1559,"thread_start_us":1423,"threads_started":1}
I20260924 10:08:54.942006 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:54.930347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:36173 (local address 127.0.0.1:42524)
0924 10:08:54.932311 (+ 1964us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:54.932332 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:54.932457 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:54.933599 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:54.933606 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:54.933631 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:54.934087 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:54.934098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.935832 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.935839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:54.938537 (+ 2698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:54.938551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.940611 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.940618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:54.940801 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:54.940829 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:08:54.940907 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:08:54.941034 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1824,"mutex_wait_us":136,"thread_start_us":1647,"threads_started":1}
I20260924 10:08:54.942413 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:54.930630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:40273 (local address 127.0.0.1:43932)
0924 10:08:54.933511 (+ 2881us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:54.933554 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:54.933707 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:54.934601 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:54.934605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:54.934630 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:54.935010 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:54.935022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.936860 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.936867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:54.938944 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:54.938954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.940274 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.940280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:54.940487 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:54.940549 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 10:08:54.940762 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 10:08:54.940895 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2721,"mutex_wait_us":3,"spinlock_wait_cycles":1280,"thread_start_us":2576,"threads_started":1}
I20260924 10:08:54.942853 16975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:54.931203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.6.2.195:36173)
0924 10:08:54.933045 (+ 1842us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:54.933055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:54.933088 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:08:54.933185 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:54.933195 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:54.933395 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:54.933604 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:54.934316 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.935700 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.938693 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:54.940096 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:54.940233 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:54.940460 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:54.941179 (+ 719us) server_negotiation.cc:299] Negotiation successful
0924 10:08:54.941338 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1671,"thread_start_us":1680,"threads_started":1}
I20260924 10:08:54.945636 16761 tablet_service.cc:1467] Processing CreateTablet for tablet 1c94b673e9e74f1fbd5a89319d22421e (DEFAULT_TABLE table=test-table [id=2f2cb5d3a92941078ade9f1b02ec69cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:54.945890 16627 tablet_service.cc:1467] Processing CreateTablet for tablet 1c94b673e9e74f1fbd5a89319d22421e (DEFAULT_TABLE table=test-table [id=2f2cb5d3a92941078ade9f1b02ec69cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:54.946116 16894 tablet_service.cc:1467] Processing CreateTablet for tablet 1c94b673e9e74f1fbd5a89319d22421e (DEFAULT_TABLE table=test-table [id=2f2cb5d3a92941078ade9f1b02ec69cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:08:54.947016 16761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c94b673e9e74f1fbd5a89319d22421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:54.947296 16627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c94b673e9e74f1fbd5a89319d22421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:54.947312 16894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c94b673e9e74f1fbd5a89319d22421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:08:54.959436 16977 tablet_bootstrap.cc:492] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040: Bootstrap starting.
I20260924 10:08:54.961971 16979 tablet_bootstrap.cc:492] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302: Bootstrap starting.
I20260924 10:08:54.963733 16978 tablet_bootstrap.cc:492] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1: Bootstrap starting.
I20260924 10:08:54.965171 16977 tablet_bootstrap.cc:654] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:54.965608 16979 tablet_bootstrap.cc:654] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:54.966450 16978 tablet_bootstrap.cc:654] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1: Neither blocks nor log segments found. Creating new log.
I20260924 10:08:54.966492 16977 log.cc:824] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:54.966692 16979 log.cc:824] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:54.967619 16978 log.cc:824] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:08:54.969413 16977 tablet_bootstrap.cc:492] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040: No bootstrap required, opened a new log
I20260924 10:08:54.969686 16977 ts_tablet_manager.cc:1397] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260924 10:08:54.970233 16978 tablet_bootstrap.cc:492] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1: No bootstrap required, opened a new log
I20260924 10:08:54.970408 16978 ts_tablet_manager.cc:1397] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 10:08:54.971413 16979 tablet_bootstrap.cc:492] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302: No bootstrap required, opened a new log
I20260924 10:08:54.971635 16979 ts_tablet_manager.cc:1397] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260924 10:08:54.977083 16978 raft_consensus.cc:348] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f870d3b5ca452fbf6e4168092e1302" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36173 } } peers { permanent_uuid: "4cc2f88ef8a14542a69b776e1194b040" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 42667 } } peers { permanent_uuid: "e51167721a9b4b09a6614a3bf622e8b1" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40273 } }
I20260924 10:08:54.977455 16978 raft_consensus.cc:374] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:54.977631 16978 raft_consensus.cc:729] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51167721a9b4b09a6614a3bf622e8b1, State: Initialized, Role: FOLLOWER
I20260924 10:08:54.978230 16978 consensus_queue.cc:260] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f870d3b5ca452fbf6e4168092e1302" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36173 } } peers { permanent_uuid: "4cc2f88ef8a14542a69b776e1194b040" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 42667 } } peers { permanent_uuid: "e51167721a9b4b09a6614a3bf622e8b1" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40273 } }
I20260924 10:08:54.980028 16978 ts_tablet_manager.cc:1428] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260924 10:08:54.980206 16977 raft_consensus.cc:348] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f870d3b5ca452fbf6e4168092e1302" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36173 } } peers { permanent_uuid: "4cc2f88ef8a14542a69b776e1194b040" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 42667 } } peers { permanent_uuid: "e51167721a9b4b09a6614a3bf622e8b1" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40273 } }
I20260924 10:08:54.980692 16977 raft_consensus.cc:374] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:54.980837 16977 raft_consensus.cc:729] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2f88ef8a14542a69b776e1194b040, State: Initialized, Role: FOLLOWER
I20260924 10:08:54.981431 16977 consensus_queue.cc:260] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f870d3b5ca452fbf6e4168092e1302" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36173 } } peers { permanent_uuid: "4cc2f88ef8a14542a69b776e1194b040" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 42667 } } peers { permanent_uuid: "e51167721a9b4b09a6614a3bf622e8b1" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40273 } }
I20260924 10:08:54.982491 16826 heartbeater.cc:503] Master 127.6.2.254:33887 was elected leader, sending a full tablet report...
I20260924 10:08:54.983143 16977 ts_tablet_manager.cc:1428] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260924 10:08:54.983038 16979 raft_consensus.cc:348] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f870d3b5ca452fbf6e4168092e1302" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36173 } } peers { permanent_uuid: "4cc2f88ef8a14542a69b776e1194b040" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 42667 } } peers { permanent_uuid: "e51167721a9b4b09a6614a3bf622e8b1" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40273 } }
I20260924 10:08:54.983512 16979 raft_consensus.cc:374] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:08:54.983615 16979 raft_consensus.cc:729] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f870d3b5ca452fbf6e4168092e1302, State: Initialized, Role: FOLLOWER
I20260924 10:08:54.984241 16979 consensus_queue.cc:260] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f870d3b5ca452fbf6e4168092e1302" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36173 } } peers { permanent_uuid: "4cc2f88ef8a14542a69b776e1194b040" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 42667 } } peers { permanent_uuid: "e51167721a9b4b09a6614a3bf622e8b1" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40273 } }
I20260924 10:08:54.986061 16979 ts_tablet_manager.cc:1428] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260924 10:08:54.986122 16959 heartbeater.cc:503] Master 127.6.2.254:33887 was elected leader, sending a full tablet report...
I20260924 10:08:55.093619 16984 raft_consensus.cc:482] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:08:55.093993 16984 raft_consensus.cc:504] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f870d3b5ca452fbf6e4168092e1302" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36173 } } peers { permanent_uuid: "4cc2f88ef8a14542a69b776e1194b040" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 42667 } } peers { permanent_uuid: "e51167721a9b4b09a6614a3bf622e8b1" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40273 } }
I20260924 10:08:55.096001 16984 leader_election.cc:288] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8f870d3b5ca452fbf6e4168092e1302 (127.6.2.195:36173), e51167721a9b4b09a6614a3bf622e8b1 (127.6.2.193:40273)
W20260924 10:08:55.099967 16827 tablet.cc:2369] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:08:55.101473 16693 tablet.cc:2369] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:08:55.105953 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:55.096665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:44945 (local address 127.6.2.193:40273)
0924 10:08:55.096952 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:55.096959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:55.097724 (+ 765us) server_negotiation.cc:407] Connection header received
0924 10:08:55.097856 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:55.097859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:55.097950 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:55.098056 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:55.098923 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.099779 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.103219 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.104609 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.104784 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:55.104902 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:55.105522 (+ 620us) server_negotiation.cc:299] Negotiation successful
0924 10:08:55.105686 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260924 10:08:55.105953 16975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:55.096432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:55863 (local address 127.6.2.195:36173)
0924 10:08:55.096712 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:55.096723 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:55.097547 (+ 824us) server_negotiation.cc:407] Connection header received
0924 10:08:55.097723 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:55.097729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:55.097838 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:55.097925 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:08:55.098677 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.099903 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.103193 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.104669 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.104800 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:55.104902 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:55.105522 (+ 620us) server_negotiation.cc:299] Negotiation successful
0924 10:08:55.105719 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 10:08:55.106233 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:55.096215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:36173 (local address 127.6.2.194:55863)
0924 10:08:55.097376 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:55.097399 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:55.097573 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:55.097968 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:55.097975 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:55.098015 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:55.098431 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:55.098441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.100027 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.100034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:55.102970 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:55.102982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.104968 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.104974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:55.105291 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:55.105345 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 10:08:55.105454 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 10:08:55.105655 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":298,"thread_start_us":670,"threads_started":1}
I20260924 10:08:55.106240 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:55.096516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:40273 (local address 127.6.2.194:44945)
0924 10:08:55.097536 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0924 10:08:55.097557 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:08:55.097725 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:08:55.098119 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:08:55.098128 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:08:55.098173 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:08:55.098697 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:55.098710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.100005 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.100012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:55.102966 (+ 2954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:08:55.102982 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.104947 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.104958 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:08:55.105291 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:55.105346 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 10:08:55.105450 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 10:08:55.105698 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":405,"thread_start_us":650,"threads_started":1}
W20260924 10:08:55.106413 16960 tablet.cc:2369] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:08:55.107546 16914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c94b673e9e74f1fbd5a89319d22421e" candidate_uuid: "4cc2f88ef8a14542a69b776e1194b040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f870d3b5ca452fbf6e4168092e1302" is_pre_election: true
I20260924 10:08:55.107926 16647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c94b673e9e74f1fbd5a89319d22421e" candidate_uuid: "4cc2f88ef8a14542a69b776e1194b040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51167721a9b4b09a6614a3bf622e8b1" is_pre_election: true
I20260924 10:08:55.108140 16914 raft_consensus.cc:2454] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2f88ef8a14542a69b776e1194b040 in term 0.
I20260924 10:08:55.108412 16647 raft_consensus.cc:2454] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2f88ef8a14542a69b776e1194b040 in term 0.
I20260924 10:08:55.108942 16717 leader_election.cc:302] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2f88ef8a14542a69b776e1194b040, a8f870d3b5ca452fbf6e4168092e1302; no voters:
I20260924 10:08:55.109366 16984 raft_consensus.cc:2781] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:08:55.109519 16984 raft_consensus.cc:482] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:08:55.109627 16984 raft_consensus.cc:3037] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:55.111974 16984 raft_consensus.cc:504] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f870d3b5ca452fbf6e4168092e1302" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36173 } } peers { permanent_uuid: "4cc2f88ef8a14542a69b776e1194b040" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 42667 } } peers { permanent_uuid: "e51167721a9b4b09a6614a3bf622e8b1" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40273 } }
I20260924 10:08:55.112597 16984 leader_election.cc:288] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [CANDIDATE]: Term 1 election: Requested vote from peers a8f870d3b5ca452fbf6e4168092e1302 (127.6.2.195:36173), e51167721a9b4b09a6614a3bf622e8b1 (127.6.2.193:40273)
I20260924 10:08:55.113113 16647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c94b673e9e74f1fbd5a89319d22421e" candidate_uuid: "4cc2f88ef8a14542a69b776e1194b040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51167721a9b4b09a6614a3bf622e8b1"
I20260924 10:08:55.113157 16914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c94b673e9e74f1fbd5a89319d22421e" candidate_uuid: "4cc2f88ef8a14542a69b776e1194b040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f870d3b5ca452fbf6e4168092e1302"
I20260924 10:08:55.113318 16647 raft_consensus.cc:3037] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:55.113358 16914 raft_consensus.cc:3037] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:08:55.115518 16647 raft_consensus.cc:2454] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2f88ef8a14542a69b776e1194b040 in term 1.
I20260924 10:08:55.115588 16914 raft_consensus.cc:2454] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2f88ef8a14542a69b776e1194b040 in term 1.
I20260924 10:08:55.116005 16714 leader_election.cc:302] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2f88ef8a14542a69b776e1194b040, e51167721a9b4b09a6614a3bf622e8b1; no voters:
I20260924 10:08:55.116382 16984 raft_consensus.cc:2781] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:08:55.116917 16984 raft_consensus.cc:686] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2f88ef8a14542a69b776e1194b040, State: Running, Role: LEADER
I20260924 10:08:55.117657 16984 consensus_queue.cc:237] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f870d3b5ca452fbf6e4168092e1302" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36173 } } peers { permanent_uuid: "4cc2f88ef8a14542a69b776e1194b040" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 42667 } } peers { permanent_uuid: "e51167721a9b4b09a6614a3bf622e8b1" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40273 } }
I20260924 10:08:55.122648 16503 catalog_manager.cc:5495] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2f88ef8a14542a69b776e1194b040 (127.6.2.194). New cstate: current_term: 1 leader_uuid: "4cc2f88ef8a14542a69b776e1194b040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f870d3b5ca452fbf6e4168092e1302" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2f88ef8a14542a69b776e1194b040" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51167721a9b4b09a6614a3bf622e8b1" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40273 } health_report { overall_health: UNKNOWN } } }
I20260924 10:08:55.179847 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:55.174233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.6.2.194:42667)
0924 10:08:55.174454 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:55.174461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:55.174563 (+ 102us) server_negotiation.cc:407] Connection header received
0924 10:08:55.174836 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:55.174840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:55.174918 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:55.175120 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:08:55.175476 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.176206 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.177543 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.177937 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.178076 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:55.178589 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:08:55.179205 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:08:55.179370 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:55.179464 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 10:08:55.179549 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:08:55.188655 16647 raft_consensus.cc:1261] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2f88ef8a14542a69b776e1194b040: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:55.188618 16914 raft_consensus.cc:1261] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2f88ef8a14542a69b776e1194b040: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:08:55.189637 16988 consensus_queue.cc:1035] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8f870d3b5ca452fbf6e4168092e1302" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:55.190017 16984 consensus_queue.cc:1035] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51167721a9b4b09a6614a3bf622e8b1" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:08:55.201087 16991 mvcc.cc:205] Tried to move back new op lower bound from 7332841616122081280 to 7332841615845945344. Current Snapshot: MvccSnapshot[applied={T|T < 7332841616122081280}]
I20260924 10:08:55.225939 16502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:55.226279 16502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:08:55.230360 16502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0f27b7095884b18b3ecdae0911011c6: Sending DeleteTablet for 3 replicas of tablet 1c94b673e9e74f1fbd5a89319d22421e
I20260924 10:08:55.231737 16627 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c94b673e9e74f1fbd5a89319d22421e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:55 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:43932
I20260924 10:08:55.231702 16761 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c94b673e9e74f1fbd5a89319d22421e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:55 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:41248
I20260924 10:08:55.231755 16894 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c94b673e9e74f1fbd5a89319d22421e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:08:55 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:42524
I20260924 10:08:55.236248 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 16564
I20260924 10:08:55.245182 17002 tablet_replica.cc:330] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302: stopping tablet replica
I20260924 10:08:55.245816 17002 raft_consensus.cc:2229] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:55.246481 17002 raft_consensus.cc:2258] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:55.249012 17002 ts_tablet_manager.cc:1905] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:55.256211 17002 ts_tablet_manager.cc:1918] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:55.256453 17002 log.cc:1196] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/1c94b673e9e74f1fbd5a89319d22421e
I20260924 10:08:55.257063 17002 ts_tablet_manager.cc:1939] T 1c94b673e9e74f1fbd5a89319d22421e P a8f870d3b5ca452fbf6e4168092e1302: Deleting consensus metadata
I20260924 10:08:55.259123 16490 catalog_manager.cc:4841] TS a8f870d3b5ca452fbf6e4168092e1302 (127.6.2.195:36173): tablet 1c94b673e9e74f1fbd5a89319d22421e (table test-table [id=2f2cb5d3a92941078ade9f1b02ec69cb]) successfully deleted
I20260924 10:08:55.260206 17000 tablet_replica.cc:330] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1: stopping tablet replica
I20260924 10:08:55.260602 17000 raft_consensus.cc:2229] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:08:55.261041 17000 raft_consensus.cc:2258] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:08:55.264608 17000 ts_tablet_manager.cc:1905] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:55.267817 17001 tablet_replica.cc:330] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040: stopping tablet replica
I20260924 10:08:55.268309 17001 raft_consensus.cc:2229] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:08:55.269016 17001 raft_consensus.cc:2258] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 10:08:55 dist-test-slave-s3n3 krb5kdc[16458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:08:55.271414 17001 ts_tablet_manager.cc:1905] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:08:55.272269 17000 ts_tablet_manager.cc:1918] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:55.272492 17000 log.cc:1196] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/1c94b673e9e74f1fbd5a89319d22421e
I20260924 10:08:55.273120 17000 ts_tablet_manager.cc:1939] T 1c94b673e9e74f1fbd5a89319d22421e P e51167721a9b4b09a6614a3bf622e8b1: Deleting consensus metadata
I20260924 10:08:55.275028 16487 catalog_manager.cc:4841] TS e51167721a9b4b09a6614a3bf622e8b1 (127.6.2.193:40273): tablet 1c94b673e9e74f1fbd5a89319d22421e (table test-table [id=2f2cb5d3a92941078ade9f1b02ec69cb]) successfully deleted
I20260924 10:08:55.278425 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:55.239913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.6.2.193:40273)
0924 10:08:55.240147 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:55.240156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:55.259421 (+ 19265us) server_negotiation.cc:407] Connection header received
0924 10:08:55.259762 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:55.259768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:55.259908 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:55.260050 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:55.261741 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.263199 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.265455 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.266131 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.266239 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:55.271499 (+ 5260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:55.271512 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:55.271529 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:55.271567 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:55.275582 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:55.276256 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:55.276265 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:55.276268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:55.276407 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:55.276828 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:55.276835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:55.276838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:55.277174 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:55.277474 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:55.277799 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 10:08:55.278001 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:08:55.279121 16687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:55.279477 17001 ts_tablet_manager.cc:1918] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:08:55.279724 17001 log.cc:1196] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/1c94b673e9e74f1fbd5a89319d22421e
I20260924 10:08:55.280354 17001 ts_tablet_manager.cc:1939] T 1c94b673e9e74f1fbd5a89319d22421e P 4cc2f88ef8a14542a69b776e1194b040: Deleting consensus metadata
I20260924 10:08:55.282085 16488 catalog_manager.cc:4841] TS 4cc2f88ef8a14542a69b776e1194b040 (127.6.2.194:42667): tablet 1c94b673e9e74f1fbd5a89319d22421e (table test-table [id=2f2cb5d3a92941078ade9f1b02ec69cb]) successfully deleted
I20260924 10:08:55.371196 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 16564
I20260924 10:08:55.390064 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 16698
Sep 24 10:08:55 dist-test-slave-s3n3 krb5kdc[16458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:08:55.406440 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:55.390946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.6.2.194:42667)
0924 10:08:55.391164 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:55.391169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:55.391190 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:08:55.391300 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:55.391305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:55.391380 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:55.391497 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:55.392663 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.393498 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.394716 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.395327 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.395457 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:55.400184 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:55.400226 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:55.400248 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:55.400298 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:55.403566 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:55.404278 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:55.404295 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:55.404297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:55.404446 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:55.404848 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:55.404852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:55.404854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:55.405219 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:55.405557 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:55.405848 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 10:08:55.406065 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:08:55.407249 16821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:55.522513 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 16698
I20260924 10:08:55.540480 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 16831
Sep 24 10:08:55 dist-test-slave-s3n3 krb5kdc[16458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:08:55.554528 16975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:08:55.541278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.6.2.195:36173)
0924 10:08:55.541428 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:08:55.541432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:08:55.541496 (+ 64us) server_negotiation.cc:407] Connection header received
0924 10:08:55.541648 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:08:55.541652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:08:55.541717 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:08:55.541803 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:08:55.543169 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.543851 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.544985 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:08:55.545340 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:08:55.545429 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:08:55.549696 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:08:55.549709 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:08:55.549725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:08:55.549763 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:08:55.552293 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:55.552883 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:55.552889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:55.552892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:55.553018 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:08:55.553326 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:08:55.553330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:08:55.553333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:08:55.553703 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:08:55.553910 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 10:08:55.554128 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 10:08:55.554274 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:08:55.555354 16954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:55.642531 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 16831
I20260924 10:08:55.659412 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 16471
I20260924 10:08:55.660511 16533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:08:55.767954 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 16471
2026-09-24T10:08:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4411 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:08:55 dist-test-slave-s3n3 krb5kdc[17010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:08:55 dist-test-slave-s3n3 krb5kdc[17010](info): set up 2 sockets
Sep 24 10:08:55 dist-test-slave-s3n3 krb5kdc[17010](info): commencing operation
krb5kdc: starting...
W20260924 10:08:56.876082 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:08:57 dist-test-slave-s3n3 krb5kdc[17010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:08:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/rangeradmin_127.6.2.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/rangeradmin_127.6.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/rangerlookup_127.6.2.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/rangerlookup_127.6.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/HTTP_127.6.2.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/HTTP_127.6.2.212@KRBTEST.COM.keytab.
I20260924 10:08:57.207512 6155 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskq3UYkq/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/postgres -l logfile start
2026-09-24 10:09:00.095 UTC [17040] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 10:09:00.095 UTC [17040] LOG: listening on IPv4 address "127.6.2.212", port 42333
2026-09-24 10:09:00.100 UTC [17040] LOG: listening on Unix socket "/tmp/.s.PGSQL.42333"
2026-09-24 10:09:00.123 UTC [17043] LOG: database system was shut down at 2026-09-24 10:08:58 UTC
2026-09-24 10:09:00.127 UTC [17040] LOG: database system is ready to accept connections
I20260924 10:09:01.127727 6155 mini_postgres.cc:96] Postgres bound to 42333
2026-09-24 10:09:01.145 UTC [17051] FATAL: database "slave" does not exist
127.6.2.212:42333 - accepting connections
I20260924 10:09:01.146215 6155 mini_ranger.cc:161] Starting Ranger...
I20260924 10:09:01.173460 6155 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 10:09:01.741942 6155 mini_ranger.cc:87] Created ranger Postgres database
I20260924 10:09:01.742091 6155 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-admin
2026-09-24 10:09:02,052 [I] DB FLAVOR :POSTGRES
2026-09-24 10:09:02,053 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:09:02,053 [I] Checking connection..
2026-09-24 10:09:02,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 10:09:02,305 [I] Checking connection passed.
2026-09-24 10:09:02,305 [I] --------- Verifying version history table ---------
2026-09-24 10:09:02,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:09:02,568 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:09:02,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:09:02,849 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:09:02,849 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 10:09:02,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 10:09:03.084 UTC [17115] WARNING: there is no transaction in progress
2026-09-24 10:09:03,091 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 10:09:03,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:09:03,358 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 10:09:03,358 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:09:03,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 10:09:03,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 10:09:03,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s3n3','N') ;"
2026-09-24 10:09:04,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:09:04,348 [I] Table x_portal_user does not exist in database ranger
2026-09-24 10:09:04,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:09:04,606 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 10:09:04,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:09:04,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:09:05,096 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 10:09:05,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 10:09:06.137 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.156 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.178 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.195 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.213 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.232 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.293 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.301 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.316 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.328 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.340 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.356 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.367 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.375 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.383 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:06.391 UTC [17223] WARNING: there is no transaction in progress
2026-09-24 10:09:07,174 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 10:09:07,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 10:09:07,416 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 10:09:07,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:09:07,688 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 10:09:07,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:09:07,934 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 10:09:07,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:09:08,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:09:08,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s3n3';"
2026-09-24 10:09:08,704 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 10:09:08,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 10:09:08,938 [I] DB_PATCHES have already been applied
I20260924 10:09:08.945349 6155 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-admin:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskq3UYkq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 10:09:08.945446 6155 mini_ranger.cc:193] Using host: 127.6.2.212
I20260924 10:09:08.960985 6155 mini_ranger.cc:227] Ranger admin URL: http://127.6.2.212:39371
Sep 24, 2026 10:09:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:09:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:09:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 10:09:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 10:09:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 10:09:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 10:09:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 10:09:10.109635 6155 mini_ranger.cc:160] Time spent starting Ranger: real 8.963s user 0.002s sys 0.039s
Sep 24, 2026 10:09:33 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 10:09:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 10:09:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 10:09:33 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 10:09:34 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 10:09:34.804700 6155 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/rangerkms_127.6.2.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/rangerkms_127.6.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.2.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.2.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.2.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/HTTP_127.6.2.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.2.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/HTTP_127.6.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 10:09:34 dist-test-slave-s3n3 krb5kdc[17010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244574, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 10:09:35.002732 6155 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 10:09:35.025565 6155 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 10:09:36.179384 6155 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 10:09:36.179615 6155 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-kms
I20260924 10:09:36.179790 6155 mini_ranger_kms.cc:227] Using postgres at 127.6.2.212:42333
2026-09-24 10:09:36,602 [I] DB FLAVOR :POSTGRES
2026-09-24 10:09:36,602 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:09:36,602 [I] Checking connection
2026-09-24 10:09:36,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 10:09:36,850 [I] connection success
2026-09-24 10:09:36,850 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 10:09:36,850 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 10:09:36,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 10:09:37,091 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 10:09:37,091 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:09:37,092 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 10:09:37,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:42333/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 10:09:37,354 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 10:09:37.879323 6155 mini_ranger_kms.cc:310] Created kms service
I20260924 10:09:38.070029 6155 mini_ranger_kms.cc:326] Created kudu user
I20260924 10:09:38.116493 6155 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 10:09:38 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244574, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:09:38.546263 6155 mini_ranger_kms.cc:384] Added ranger policy
I20260924 10:09:38.546602 6155 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-kms:/tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 10:09:38.546666 6155 mini_ranger_kms.cc:240] Using host: 127.6.2.212
I20260924 10:09:38.573840 6155 mini_ranger_kms.cc:276] Ranger KMS PID: 17404
I20260924 10:09:38.574054 6155 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.2.212:60859
Sep 24, 2026 10:09:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:09:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:09:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 10:09:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 10:09:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 10:09:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 10:09:40.921607 6155 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.919s user 0.118s sys 0.329s
I20260924 10:09:40.921859 6155 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 10:09:40.922052 6155 mini_ranger_kms.cc:401] 127.6.2.212:60859/kms/v1/keys
Sep 24 10:09:41 dist-test-slave-s3n3 krb5kdc[17010](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790244581, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.2.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 10:09:41 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244581, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.2.212@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
Sep 24 10:09:41 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244581, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.2.212@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
Sep 24 10:09:44 dist-test-slave-s3n3 krb5kdc[17010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:09:44.135039 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:33173
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:33173
--ranger_config_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.2.212:60859/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:09:44.480242 17454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:09:44.480557 17454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:09:44.480711 17454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:09:44.488431 17454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:09:44.488546 17454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:09:44.488628 17454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:09:44.488713 17454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:09:44.488823 17454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:09:44.499084 17454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.2.212:60859/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:33173
--ranger_config_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:33173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:09:44.500460 17454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:09:44.501734 17454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:09:44.514051 17459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:09:44.513981 17462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:09:44.514017 17460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:09:44.514956 17454 server_base.cc:1034] running on GCE node
I20260924 10:09:44.515795 17454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:09:44.517611 17454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:09:44.518849 17454 hybrid_clock.cc:648] HybridClock initialized: now 1790244584518803 us; error 37 us; skew 500 ppm
I20260924 10:09:44.519260 17454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:09:44 dist-test-slave-s3n3 krb5kdc[17010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244584, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:09:44.527118 17454 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:09:44.528534 17454 webserver.cc:460] Webserver started at http://127.6.2.254:40293/ using document root <none> and password file <none>
I20260924 10:09:44.529145 17454 fs_manager.cc:359] Metadata directory not provided
I20260924 10:09:44.529238 17454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:09:44.529472 17454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:09:44 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244584, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:09:44.594897 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "6b7c3e6e1827485a8275b1bf5e67bc9f"
format_stamp: "Formatted at 2026-09-24 10:09:44 on dist-test-slave-s3n3"
server_key: "07ae970d0fdbd1414bfe5ac86828d0f6"
server_key_iv: "a1ac64bbc169c36412e638429d86f549"
server_key_version: "kuduclusterkey@0"
I20260924 10:09:44.595904 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "6b7c3e6e1827485a8275b1bf5e67bc9f"
format_stamp: "Formatted at 2026-09-24 10:09:44 on dist-test-slave-s3n3"
server_key: "07ae970d0fdbd1414bfe5ac86828d0f6"
server_key_iv: "a1ac64bbc169c36412e638429d86f549"
server_key_version: "kuduclusterkey@0"
I20260924 10:09:44.601568 17454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:09:44.619179 17469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:09:44.620906 17454 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 10:09:44.621125 17454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "6b7c3e6e1827485a8275b1bf5e67bc9f"
format_stamp: "Formatted at 2026-09-24 10:09:44 on dist-test-slave-s3n3"
server_key: "07ae970d0fdbd1414bfe5ac86828d0f6"
server_key_iv: "a1ac64bbc169c36412e638429d86f549"
server_key_version: "kuduclusterkey@0"
I20260924 10:09:44.621367 17454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:09:44.644928 17454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:09:44.649184 17454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:09:44.649523 17454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:09:44.669831 17454 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:33173
I20260924 10:09:44.669878 17520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:33173 every 8 connection(s)
I20260924 10:09:44.671408 17454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:09:44.673039 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 17454
I20260924 10:09:44.673259 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:09:44.673885 6155 external_mini_cluster.cc:1428] Setting key 2d84bd2725f1fb6b61d470e24202fadc
I20260924 10:09:44.676199 17521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 10:09:44 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:09:44.689925 17521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f: Bootstrap starting.
I20260924 10:09:44.694111 17521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f: Neither blocks nor log segments found. Creating new log.
I20260924 10:09:44.695361 17521 log.cc:824] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:09:44.697890 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:44.675790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.6.2.254:33173)
0924 10:09:44.676542 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:44.676553 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:44.676591 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:09:44.678325 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:44.678345 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:44.678707 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:44.679024 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:09:44.680720 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:44.682072 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:44.683777 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:44.684418 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:44.684548 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:44.689532 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:09:44.689544 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:09:44.689567 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:09:44.689612 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:09:44.693396 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:44.694090 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:44.694104 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:44.694113 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:44.694316 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:44.694689 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:44.694693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:44.694696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:44.695390 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:09:44.695714 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:44.696200 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 10:09:44.696593 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":351,"threads_started":1}
I20260924 10:09:44.698735 17521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f: No bootstrap required, opened a new log
I20260924 10:09:44.706842 17521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c3e6e1827485a8275b1bf5e67bc9f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33173 } }
I20260924 10:09:44.707331 17521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:09:44.707458 17521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7c3e6e1827485a8275b1bf5e67bc9f, State: Initialized, Role: FOLLOWER
I20260924 10:09:44.708045 17521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c3e6e1827485a8275b1bf5e67bc9f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33173 } }
I20260924 10:09:44.708242 17521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:09:44.708338 17521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:09:44.708464 17521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:09:44.710834 17521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c3e6e1827485a8275b1bf5e67bc9f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33173 } }
I20260924 10:09:44.711369 17521 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b7c3e6e1827485a8275b1bf5e67bc9f; no voters:
I20260924 10:09:44.711892 17521 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:09:44.712044 17526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:09:44.712819 17526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [term 1 LEADER]: Becoming Leader. State: Replica: 6b7c3e6e1827485a8275b1bf5e67bc9f, State: Running, Role: LEADER
I20260924 10:09:44.713413 17526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c3e6e1827485a8275b1bf5e67bc9f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33173 } }
I20260924 10:09:44.714182 17521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:09:44.716308 17528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b7c3e6e1827485a8275b1bf5e67bc9f. Latest consensus state: current_term: 1 leader_uuid: "6b7c3e6e1827485a8275b1bf5e67bc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c3e6e1827485a8275b1bf5e67bc9f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33173 } } }
I20260924 10:09:44.716711 17528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [sys.catalog]: This master's current role is: LEADER
I20260924 10:09:44.716428 17527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b7c3e6e1827485a8275b1bf5e67bc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c3e6e1827485a8275b1bf5e67bc9f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33173 } } }
I20260924 10:09:44.716905 17527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f [sys.catalog]: This master's current role is: LEADER
I20260924 10:09:44.717653 17532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:09:44.724035 17532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:09:44.725025 17521 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 10:09:44.742157 17532 catalog_manager.cc:1347] Generated new cluster ID: f2ec5df8cedf4a0abe5b4e9bc3cd9448
I20260924 10:09:44.742300 17532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:09:44.771939 17532 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:09:44.774497 17532 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:09:44.788434 17532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f: Generated new TSK 0
I20260924 10:09:44.789644 17532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:09:45.576550 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.2.212:60859/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33173
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:09:45 dist-test-slave-s3n3 krb5kdc[17010](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790244585, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 10:09:46.026708 17566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:09:46.027144 17566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:09:46.027307 17566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:09:46.039527 17566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:09:46.039670 17566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:09:46.039904 17566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:09:46.052718 17566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.2.212:60859/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:09:46.054088 17566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:09:46.055456 17566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:09:46.073602 17576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:09:46.074602 17577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:09:46.077513 17566 server_base.cc:1034] running on GCE node
W20260924 10:09:46.083253 17579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:09:46.084121 17566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:09:46.085762 17566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:09:46.087046 17566 hybrid_clock.cc:648] HybridClock initialized: now 1790244586086981 us; error 62 us; skew 500 ppm
I20260924 10:09:46.087446 17566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:09:46 dist-test-slave-s3n3 krb5kdc[17010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244586, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:09:46.092137 17566 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:09:46.095376 17566 webserver.cc:460] Webserver started at http://127.6.2.193:42161/ using document root <none> and password file <none>
I20260924 10:09:46.096119 17566 fs_manager.cc:359] Metadata directory not provided
I20260924 10:09:46.096269 17566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:09:46.096611 17566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:09:46 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244586, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:09:46.119843 17566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef"
format_stamp: "Formatted at 2026-09-24 10:09:46 on dist-test-slave-s3n3"
server_key: "145cece4b3750cf0cf4156523523e0b3"
server_key_iv: "4da1db804838d6f6a98e257f0c4ba5fa"
server_key_version: "kuduclusterkey@0"
I20260924 10:09:46.121096 17566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef"
format_stamp: "Formatted at 2026-09-24 10:09:46 on dist-test-slave-s3n3"
server_key: "145cece4b3750cf0cf4156523523e0b3"
server_key_iv: "4da1db804838d6f6a98e257f0c4ba5fa"
server_key_version: "kuduclusterkey@0"
I20260924 10:09:46.129285 17566 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260924 10:09:46.148342 17586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:09:46.150349 17566 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:09:46.150612 17566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef"
format_stamp: "Formatted at 2026-09-24 10:09:46 on dist-test-slave-s3n3"
server_key: "145cece4b3750cf0cf4156523523e0b3"
server_key_iv: "4da1db804838d6f6a98e257f0c4ba5fa"
server_key_version: "kuduclusterkey@0"
I20260924 10:09:46.150980 17566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:09:46.202700 17566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:09:46.208423 17566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:09:46.209316 17566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:09:46.210700 17566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:09:46.212455 17566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:09:46.212553 17566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:09:46.212675 17566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:09:46.212751 17566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:09:46 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244585, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:09:46.335521 17566 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:43191
I20260924 10:09:46.335498 17698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:43191 every 8 connection(s)
I20260924 10:09:46.337656 17566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:09:46.339751 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 17566
I20260924 10:09:46.339967 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:09:46.340850 6155 external_mini_cluster.cc:1428] Setting key 3e76c6ce995f26dae56b7c781f09ca99
Sep 24 10:09:46 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244586, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
I20260924 10:09:46.402499 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:46.354327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:42905 (local address 127.6.2.254:33173)
0924 10:09:46.368958 (+ 14631us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:46.368966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:46.372836 (+ 3870us) server_negotiation.cc:407] Connection header received
0924 10:09:46.372926 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:46.372935 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:46.373045 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:46.373151 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:09:46.375531 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:46.376763 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:46.379827 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:46.380599 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:46.380716 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:46.385830 (+ 5114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:09:46.385837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:09:46.385843 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:09:46.385888 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:09:46.389034 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:46.389909 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:46.389914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:46.389917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:46.390074 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:46.390654 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:46.390659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:46.390662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:46.391046 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:09:46.399144 (+ 8098us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:46.401832 (+ 2688us) server_negotiation.cc:299] Negotiation successful
0924 10:09:46.402102 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14463,"thread_start_us":14334,"threads_started":1}
I20260924 10:09:46.405364 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:46.354312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:33173 (local address 127.6.2.193:42905)
0924 10:09:46.369436 (+ 15124us) negotiation.cc:107] Waiting for socket to connect
0924 10:09:46.369503 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 10:09:46.372371 (+ 2868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:09:46.373649 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:09:46.373664 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:09:46.374371 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:09:46.375302 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:46.375330 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:46.376995 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:46.377000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:46.379644 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:46.379662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:46.381084 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:46.381092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:46.381532 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:46.382425 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:09:46.382462 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:09:46.385528 (+ 3066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:09:46.389236 (+ 3708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:09:46.389246 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:09:46.389265 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:09:46.389784 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:09:46.390229 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:09:46.390236 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:09:46.390243 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:09:46.390516 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:09:46.399423 (+ 8907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:09:46.399434 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:09:46.399967 (+ 533us) client_negotiation.cc:769] Sending connection context
0924 10:09:46.403310 (+ 3343us) client_negotiation.cc:240] Negotiation successful
0924 10:09:46.403914 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14739,"thread_start_us":23931,"threads_started":1}
I20260924 10:09:46.407199 17699 heartbeater.cc:347] Connected to a master server at 127.6.2.254:33173
I20260924 10:09:46.407542 17699 heartbeater.cc:464] Registering TS with master...
I20260924 10:09:46.408411 17699 heartbeater.cc:511] Master 127.6.2.254:33173 requested a full tablet report, sending...
I20260924 10:09:46.410863 17486 ts_manager.cc:194] Registered new tserver with Master: 31a02d8fd1fa4c498a19cb35be2fd9ef (127.6.2.193:43191)
I20260924 10:09:46.413051 17486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:42905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:09:46.431250 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.2.212:60859/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33173
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:09:46 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244585, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
W20260924 10:09:46.918818 17705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:09:46.919384 17705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:09:46.919588 17705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:09:46.931797 17705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:09:46.932116 17705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:09:46.932337 17705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:09:46.948107 17705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.2.212:60859/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:09:46.950114 17705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:09:46.951952 17705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:09:46.986743 17714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:09:46.990195 17712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:09:46.993490 17718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:09:46.994086 17705 server_base.cc:1034] running on GCE node
I20260924 10:09:46.995102 17705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:09:46.999636 17705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:09:47.000972 17705 hybrid_clock.cc:648] HybridClock initialized: now 1790244587000863 us; error 107 us; skew 500 ppm
I20260924 10:09:47.001434 17705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:09:46 dist-test-slave-s3n3 krb5kdc[17010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244586, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:09:47.006024 17705 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:09:47.008055 17705 webserver.cc:460] Webserver started at http://127.6.2.194:44547/ using document root <none> and password file <none>
I20260924 10:09:47.008829 17705 fs_manager.cc:359] Metadata directory not provided
I20260924 10:09:47.008975 17705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:09:47.009358 17705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:09:47 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244586, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:09:47.027900 17705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "a4b985106a5a4a1eb18d158165741c48"
format_stamp: "Formatted at 2026-09-24 10:09:47 on dist-test-slave-s3n3"
server_key: "4f7277d80274f1614144573253e2c7b5"
server_key_iv: "390a489296e9effd903479bb5aaf0c32"
server_key_version: "kuduclusterkey@0"
I20260924 10:09:47.029349 17705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "a4b985106a5a4a1eb18d158165741c48"
format_stamp: "Formatted at 2026-09-24 10:09:47 on dist-test-slave-s3n3"
server_key: "4f7277d80274f1614144573253e2c7b5"
server_key_iv: "390a489296e9effd903479bb5aaf0c32"
server_key_version: "kuduclusterkey@0"
I20260924 10:09:47.035810 17705 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 10:09:47.086611 17729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:09:47.092870 17705 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.010s sys 0.005s
I20260924 10:09:47.093199 17705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "a4b985106a5a4a1eb18d158165741c48"
format_stamp: "Formatted at 2026-09-24 10:09:47 on dist-test-slave-s3n3"
server_key: "4f7277d80274f1614144573253e2c7b5"
server_key_iv: "390a489296e9effd903479bb5aaf0c32"
server_key_version: "kuduclusterkey@0"
I20260924 10:09:47.093583 17705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:09:47.169551 17705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:09:47.174953 17705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:09:47.175299 17705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:09:47.176476 17705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:09:47.178263 17705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:09:47.178354 17705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:09:47.178529 17705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:09:47.178594 17705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:09:47.225244 17705 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:44207
I20260924 10:09:47.225310 17841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:44207 every 8 connection(s)
I20260924 10:09:47.226840 17705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:09:47.234606 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 17705
I20260924 10:09:47.234797 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:09:47.235414 6155 external_mini_cluster.cc:1428] Setting key 65585df2285edb4b6b6e7d1879c8ed9f
Sep 24 10:09:47 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244586, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:09:47.263809 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:47.231314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:36843 (local address 127.6.2.254:33173)
0924 10:09:47.233347 (+ 2033us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:47.233354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:47.237372 (+ 4018us) server_negotiation.cc:407] Connection header received
0924 10:09:47.240185 (+ 2813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:47.240194 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:47.240295 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:47.240410 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:09:47.242931 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.244341 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.247330 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.248144 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.248270 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:47.253582 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:09:47.253592 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:09:47.253600 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:09:47.253651 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:09:47.258535 (+ 4884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:47.259793 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:47.259802 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:47.259809 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:47.259985 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:47.260627 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:47.260635 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:47.260640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:47.261560 (+ 920us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:09:47.261743 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:47.263021 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0924 10:09:47.263331 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1888,"thread_start_us":1757,"threads_started":1}
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:09:47.267051 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:47.230296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:33173 (local address 127.6.2.194:36843)
0924 10:09:47.236935 (+ 6639us) negotiation.cc:107] Waiting for socket to connect
0924 10:09:47.236997 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 10:09:47.239695 (+ 2698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:09:47.240965 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:09:47.240981 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:09:47.241737 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:09:47.242695 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:47.242726 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.244552 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.244561 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:47.247118 (+ 2557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:47.247147 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.248650 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.248659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:47.249039 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:47.249966 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:09:47.250005 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:09:47.253291 (+ 3286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:09:47.258799 (+ 5508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:09:47.258813 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:09:47.258840 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:09:47.259613 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:09:47.260168 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:09:47.260174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:09:47.260178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:09:47.260475 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:09:47.261867 (+ 1392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:09:47.261881 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:09:47.262456 (+ 575us) client_negotiation.cc:769] Sending connection context
0924 10:09:47.264489 (+ 2033us) client_negotiation.cc:240] Negotiation successful
0924 10:09:47.265175 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6323,"thread_start_us":6095,"threads_started":1}
I20260924 10:09:47.268931 17842 heartbeater.cc:347] Connected to a master server at 127.6.2.254:33173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
I20260924 10:09:47.269279 17842 heartbeater.cc:464] Registering TS with master...
I20260924 10:09:47.270339 17842 heartbeater.cc:511] Master 127.6.2.254:33173 requested a full tablet report, sending...
I20260924 10:09:47.275913 17486 ts_manager.cc:194] Registered new tserver with Master: a4b985106a5a4a1eb18d158165741c48 (127.6.2.194:44207)
I20260924 10:09:47.277328 17486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:36843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:09:47.302367 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.2.212:60859/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33173
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:09:47.416970 17699 heartbeater.cc:503] Master 127.6.2.254:33173 was elected leader, sending a full tablet report...
W20260924 10:09:47.650357 17848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:09:47.650640 17848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:09:47.650730 17848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:09:47.658648 17848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:09:47.658764 17848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:09:47.658876 17848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:09:47.668908 17848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.2.212:60859/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:09:47.670166 17848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:09:47.671504 17848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:09:47.684803 17853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:09:47.684775 17854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:09:47.685007 17856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:09:47.685982 17848 server_base.cc:1034] running on GCE node
I20260924 10:09:47.686592 17848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:09:47.687685 17848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:09:47.688910 17848 hybrid_clock.cc:648] HybridClock initialized: now 1790244587688839 us; error 41 us; skew 500 ppm
I20260924 10:09:47.689250 17848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:09:47 dist-test-slave-s3n3 krb5kdc[17010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244587, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:09:47.692536 17848 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:09:47.694761 17848 webserver.cc:460] Webserver started at http://127.6.2.195:44773/ using document root <none> and password file <none>
I20260924 10:09:47.695524 17848 fs_manager.cc:359] Metadata directory not provided
I20260924 10:09:47.695653 17848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:09:47.695983 17848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:09:47 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244587, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:09:47.713156 17848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "1954a3594b794a968dbbc2ac1e788c06"
format_stamp: "Formatted at 2026-09-24 10:09:47 on dist-test-slave-s3n3"
server_key: "274ce99c462e27c270ed4ab8322a98aa"
server_key_iv: "7ec8f391069e81d6315fb494383fc85f"
server_key_version: "kuduclusterkey@0"
I20260924 10:09:47.714213 17848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "1954a3594b794a968dbbc2ac1e788c06"
format_stamp: "Formatted at 2026-09-24 10:09:47 on dist-test-slave-s3n3"
server_key: "274ce99c462e27c270ed4ab8322a98aa"
server_key_iv: "7ec8f391069e81d6315fb494383fc85f"
server_key_version: "kuduclusterkey@0"
I20260924 10:09:47.720080 17848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:09:47.737980 17863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:09:47.739693 17848 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 10:09:47.739918 17848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "1954a3594b794a968dbbc2ac1e788c06"
format_stamp: "Formatted at 2026-09-24 10:09:47 on dist-test-slave-s3n3"
server_key: "274ce99c462e27c270ed4ab8322a98aa"
server_key_iv: "7ec8f391069e81d6315fb494383fc85f"
server_key_version: "kuduclusterkey@0"
I20260924 10:09:47.740164 17848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:09:47.765337 17848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:09:47.769881 17848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:09:47.770215 17848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:09:47.771220 17848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:09:47.772634 17848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:09:47.772742 17848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:09:47.772928 17848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:09:47.773006 17848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:09:47.818720 17848 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:42623
I20260924 10:09:47.818809 17975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:42623 every 8 connection(s)
I20260924 10:09:47.820282 17848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:09:47.827697 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 17848
I20260924 10:09:47.827907 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:09:47.828540 6155 external_mini_cluster.cc:1428] Setting key 0d66c3b66c040de85ac760921800b280
Sep 24 10:09:47 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244587, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:09:47.846304 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:47.822971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:46591 (local address 127.6.2.254:33173)
0924 10:09:47.823652 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:47.823660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:47.825288 (+ 1628us) server_negotiation.cc:407] Connection header received
0924 10:09:47.830695 (+ 5407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:47.830701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:47.830857 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:47.831029 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:09:47.833182 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.834166 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.835992 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.836448 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.836569 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:47.840265 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:09:47.840271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:09:47.840274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:09:47.840306 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:09:47.842905 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:47.843734 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:47.843739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:47.843742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:47.843901 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:47.844553 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:47.844558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:47.844561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:47.844933 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:09:47.845103 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:47.845729 (+ 626us) server_negotiation.cc:299] Negotiation successful
0924 10:09:47.845945 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":320,"threads_started":1}
I20260924 10:09:47.847385 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:47.823651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:33173 (local address 127.6.2.195:46591)
0924 10:09:47.825066 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0924 10:09:47.825130 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 10:09:47.830199 (+ 5069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:09:47.831410 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:09:47.831433 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:09:47.832191 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:09:47.832971 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:47.832990 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.834385 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.834390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:47.835821 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:47.835833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.836701 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.836707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:47.836925 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:47.837598 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:09:47.837628 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:09:47.840016 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:09:47.843119 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:09:47.843131 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:09:47.843150 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:09:47.843597 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:09:47.844131 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:09:47.844137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:09:47.844140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:09:47.844422 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:09:47.845058 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:09:47.845066 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:09:47.845390 (+ 324us) client_negotiation.cc:769] Sending connection context
0924 10:09:47.845649 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 10:09:47.846145 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"thread_start_us":837,"threads_started":1}
I20260924 10:09:47.848986 17976 heartbeater.cc:347] Connected to a master server at 127.6.2.254:33173
I20260924 10:09:47.849232 17976 heartbeater.cc:464] Registering TS with master...
I20260924 10:09:47.849844 17976 heartbeater.cc:511] Master 127.6.2.254:33173 requested a full tablet report, sending...
I20260924 10:09:47.851675 17486 ts_manager.cc:194] Registered new tserver with Master: 1954a3594b794a968dbbc2ac1e788c06 (127.6.2.195:42623)
I20260924 10:09:47.852421 17486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:46591
I20260924 10:09:47.858404 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:09:47.876904 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:47.863268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.6.2.254:33173)
0924 10:09:47.863433 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:47.863438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:47.863996 (+ 558us) server_negotiation.cc:407] Connection header received
0924 10:09:47.864219 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:47.864225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:47.864334 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:47.864455 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:09:47.865966 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.866788 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.868393 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.868860 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.868997 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:47.871613 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:09:47.871618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:09:47.871622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:09:47.871653 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:09:47.874230 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:47.874927 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:47.874933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:47.874937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:47.875109 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:47.875500 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:47.875505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:47.875508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:47.875775 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:09:47.876024 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:47.876365 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 10:09:47.876597 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:09:47.881227 17486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:09:47.888810 17486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:09:47.918725 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:47.903563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.6.2.194:44207)
0924 10:09:47.906066 (+ 2503us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:47.906078 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:47.906111 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:09:47.906208 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:47.906216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:47.906499 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:47.907595 (+ 1096us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:09:47.907908 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.910073 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.914041 (+ 3968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.915705 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.917506 (+ 1801us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:47.917765 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:47.918161 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 10:09:47.918343 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2303,"thread_start_us":2168,"threads_started":1}
I20260924 10:09:47.918744 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:47.903481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:44207 (local address 127.0.0.1:35640)
0924 10:09:47.905386 (+ 1905us) negotiation.cc:107] Waiting for socket to connect
0924 10:09:47.905401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:09:47.905513 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:09:47.906746 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:09:47.906754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:09:47.906782 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:09:47.907310 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:47.907327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.910241 (+ 2914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.910248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:47.913703 (+ 3455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:47.913726 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.917591 (+ 3865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.917601 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:47.917904 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:47.917971 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 10:09:47.918108 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 10:09:47.918320 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1811,"thread_start_us":854,"threads_started":1}
I20260924 10:09:47.923308 17777 tablet_service.cc:1467] Processing CreateTablet for tablet f2a8e5b7686a498db937a80bba2bdc61 (DEFAULT_TABLE table=test-table [id=39a276ce2c8d43fe9f8a0f9b2aa48ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:09:47.925005 17777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a8e5b7686a498db937a80bba2bdc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:09:47.925539 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:47.905912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.6.2.195:42623)
0924 10:09:47.916952 (+ 11040us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:47.916965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:47.917767 (+ 802us) server_negotiation.cc:407] Connection header received
0924 10:09:47.917894 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:47.917902 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:47.918138 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:47.918327 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:09:47.919332 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.921069 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.923397 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.924387 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.924526 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:47.924755 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:47.924973 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 10:09:47.925158 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10803,"thread_start_us":10606,"threads_started":1}
I20260924 10:09:47.925805 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:47.905591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:42623 (local address 127.0.0.1:35206)
0924 10:09:47.917446 (+ 11855us) negotiation.cc:107] Waiting for socket to connect
0924 10:09:47.917467 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:09:47.917605 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:09:47.918600 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:09:47.918605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:09:47.918629 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:09:47.919120 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:47.919133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.921424 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.921431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:47.923254 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:47.923265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.924580 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.924586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:47.924751 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:47.924796 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 10:09:47.924902 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 10:09:47.925051 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11718,"mutex_wait_us":3,"thread_start_us":11577,"threads_started":1}
I20260924 10:09:47.926370 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:47.902603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:43191 (local address 127.0.0.1:41704)
0924 10:09:47.904763 (+ 2160us) negotiation.cc:107] Waiting for socket to connect
0924 10:09:47.904812 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:09:47.905010 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:09:47.917145 (+ 12135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:09:47.917151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:09:47.917179 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:09:47.917774 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:47.917787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.919957 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.919961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:47.922624 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:47.922639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.924957 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.924963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:47.925125 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:47.925152 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:09:47.925246 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 10:09:47.926000 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1898,"thread_start_us":2545,"threads_started":1}
I20260924 10:09:47.926628 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:47.902879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.6.2.193:43191)
0924 10:09:47.916494 (+ 13615us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:47.916509 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:47.916543 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:09:47.916652 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:47.916661 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:47.916931 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:47.917162 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:09:47.917959 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.919784 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.922806 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:47.924398 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:47.924543 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:47.924782 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:47.925499 (+ 717us) server_negotiation.cc:299] Negotiation successful
0924 10:09:47.925687 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13354,"thread_start_us":13130,"threads_started":1}
I20260924 10:09:47.929275 17911 tablet_service.cc:1467] Processing CreateTablet for tablet f2a8e5b7686a498db937a80bba2bdc61 (DEFAULT_TABLE table=test-table [id=39a276ce2c8d43fe9f8a0f9b2aa48ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:09:47.930467 17911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a8e5b7686a498db937a80bba2bdc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:09:47.930322 17634 tablet_service.cc:1467] Processing CreateTablet for tablet f2a8e5b7686a498db937a80bba2bdc61 (DEFAULT_TABLE table=test-table [id=39a276ce2c8d43fe9f8a0f9b2aa48ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:09:47.932256 17634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a8e5b7686a498db937a80bba2bdc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:09:47.941699 17995 tablet_bootstrap.cc:492] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06: Bootstrap starting.
I20260924 10:09:47.943790 17996 tablet_bootstrap.cc:492] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48: Bootstrap starting.
I20260924 10:09:47.946341 17995 tablet_bootstrap.cc:654] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06: Neither blocks nor log segments found. Creating new log.
I20260924 10:09:47.947458 17996 tablet_bootstrap.cc:654] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48: Neither blocks nor log segments found. Creating new log.
I20260924 10:09:47.948091 17995 log.cc:824] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06: Log is configured to *not* fsync() on all Append() calls
I20260924 10:09:47.948932 17997 tablet_bootstrap.cc:492] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef: Bootstrap starting.
I20260924 10:09:47.949195 17996 log.cc:824] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48: Log is configured to *not* fsync() on all Append() calls
I20260924 10:09:47.951967 17995 tablet_bootstrap.cc:492] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06: No bootstrap required, opened a new log
I20260924 10:09:47.952226 17995 ts_tablet_manager.cc:1397] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 10:09:47.952593 17996 tablet_bootstrap.cc:492] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48: No bootstrap required, opened a new log
I20260924 10:09:47.952869 17997 tablet_bootstrap.cc:654] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef: Neither blocks nor log segments found. Creating new log.
I20260924 10:09:47.952873 17996 ts_tablet_manager.cc:1397] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 10:09:47.954231 17997 log.cc:824] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef: Log is configured to *not* fsync() on all Append() calls
I20260924 10:09:47.958323 17997 tablet_bootstrap.cc:492] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef: No bootstrap required, opened a new log
I20260924 10:09:47.958585 17997 ts_tablet_manager.cc:1397] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 10:09:47.962437 17996 raft_consensus.cc:348] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43191 } } peers { permanent_uuid: "1954a3594b794a968dbbc2ac1e788c06" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42623 } } peers { permanent_uuid: "a4b985106a5a4a1eb18d158165741c48" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44207 } }
I20260924 10:09:47.962479 17995 raft_consensus.cc:348] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43191 } } peers { permanent_uuid: "1954a3594b794a968dbbc2ac1e788c06" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42623 } } peers { permanent_uuid: "a4b985106a5a4a1eb18d158165741c48" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44207 } }
I20260924 10:09:47.962816 17996 raft_consensus.cc:374] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:09:47.962888 17995 raft_consensus.cc:374] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:09:47.962914 17996 raft_consensus.cc:729] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b985106a5a4a1eb18d158165741c48, State: Initialized, Role: FOLLOWER
I20260924 10:09:47.962992 17995 raft_consensus.cc:729] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1954a3594b794a968dbbc2ac1e788c06, State: Initialized, Role: FOLLOWER
I20260924 10:09:47.963586 17996 consensus_queue.cc:260] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43191 } } peers { permanent_uuid: "1954a3594b794a968dbbc2ac1e788c06" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42623 } } peers { permanent_uuid: "a4b985106a5a4a1eb18d158165741c48" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44207 } }
I20260924 10:09:47.963711 17995 consensus_queue.cc:260] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43191 } } peers { permanent_uuid: "1954a3594b794a968dbbc2ac1e788c06" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42623 } } peers { permanent_uuid: "a4b985106a5a4a1eb18d158165741c48" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44207 } }
I20260924 10:09:47.964722 17842 heartbeater.cc:503] Master 127.6.2.254:33173 was elected leader, sending a full tablet report...
I20260924 10:09:47.965262 17996 ts_tablet_manager.cc:1428] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 10:09:47.966543 17976 heartbeater.cc:503] Master 127.6.2.254:33173 was elected leader, sending a full tablet report...
I20260924 10:09:47.967271 17995 ts_tablet_manager.cc:1428] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20260924 10:09:47.969244 17997 raft_consensus.cc:348] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43191 } } peers { permanent_uuid: "1954a3594b794a968dbbc2ac1e788c06" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42623 } } peers { permanent_uuid: "a4b985106a5a4a1eb18d158165741c48" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44207 } }
I20260924 10:09:47.969795 17997 raft_consensus.cc:374] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:09:47.969923 17997 raft_consensus.cc:729] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a02d8fd1fa4c498a19cb35be2fd9ef, State: Initialized, Role: FOLLOWER
I20260924 10:09:47.970539 17997 consensus_queue.cc:260] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43191 } } peers { permanent_uuid: "1954a3594b794a968dbbc2ac1e788c06" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42623 } } peers { permanent_uuid: "a4b985106a5a4a1eb18d158165741c48" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44207 } }
I20260924 10:09:47.972317 17997 ts_tablet_manager.cc:1428] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
W20260924 10:09:47.985275 17843 tablet.cc:2369] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:09:48.072046 17977 tablet.cc:2369] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:09:48.103585 17700 tablet.cc:2369] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:09:48.266626 18003 raft_consensus.cc:482] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:09:48.266956 18003 raft_consensus.cc:504] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43191 } } peers { permanent_uuid: "1954a3594b794a968dbbc2ac1e788c06" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42623 } } peers { permanent_uuid: "a4b985106a5a4a1eb18d158165741c48" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44207 } }
I20260924 10:09:48.268769 18003 leader_election.cc:288] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1954a3594b794a968dbbc2ac1e788c06 (127.6.2.195:42623), a4b985106a5a4a1eb18d158165741c48 (127.6.2.194:44207)
I20260924 10:09:48.276929 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:48.269100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:42623 (local address 127.6.2.193:33375)
0924 10:09:48.270024 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0924 10:09:48.270050 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:09:48.270215 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:09:48.270707 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:09:48.270712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:09:48.270740 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:09:48.271181 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:48.271197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.272478 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.272484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:48.274624 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:48.274636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.276135 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.276142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:48.276322 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:48.276364 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 10:09:48.276478 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 10:09:48.276626 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":1482,"threads_started":1}
I20260924 10:09:48.276924 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:48.269218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:33375 (local address 127.6.2.195:42623)
0924 10:09:48.269467 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:48.269477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:48.270151 (+ 674us) server_negotiation.cc:407] Connection header received
0924 10:09:48.270353 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:48.270364 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:48.270422 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:48.270522 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:09:48.271368 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.272344 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.274799 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.275932 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.276032 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:48.276121 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:48.276493 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 10:09:48.276650 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:09:48.277417 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:48.269222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:44207 (local address 127.6.2.193:47027)
0924 10:09:48.270812 (+ 1590us) negotiation.cc:107] Waiting for socket to connect
0924 10:09:48.270826 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:09:48.271149 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:09:48.271505 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:09:48.271510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:09:48.271535 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:09:48.271875 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:48.271886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.273636 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.273641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:48.275311 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:09:48.275326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.276648 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.276654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:09:48.276815 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:48.276843 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:09:48.276947 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 10:09:48.277086 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1461,"thread_start_us":618,"threads_started":1}
I20260924 10:09:48.277690 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:48.269376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:47027 (local address 127.6.2.194:44207)
0924 10:09:48.269713 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:48.269722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:48.270919 (+ 1197us) server_negotiation.cc:407] Connection header received
0924 10:09:48.271294 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:48.271301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:48.271386 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:48.271461 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:09:48.272048 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.273440 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.275477 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.276434 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.276558 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:48.276644 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:48.276976 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 10:09:48.277430 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260924 10:09:48.278064 17931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a8e5b7686a498db937a80bba2bdc61" candidate_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1954a3594b794a968dbbc2ac1e788c06" is_pre_election: true
I20260924 10:09:48.278613 17931 raft_consensus.cc:2454] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a02d8fd1fa4c498a19cb35be2fd9ef in term 0.
I20260924 10:09:48.278600 17797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a8e5b7686a498db937a80bba2bdc61" candidate_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b985106a5a4a1eb18d158165741c48" is_pre_election: true
I20260924 10:09:48.279145 17797 raft_consensus.cc:2454] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a02d8fd1fa4c498a19cb35be2fd9ef in term 0.
I20260924 10:09:48.279371 17590 leader_election.cc:302] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1954a3594b794a968dbbc2ac1e788c06, 31a02d8fd1fa4c498a19cb35be2fd9ef; no voters:
I20260924 10:09:48.279779 18003 raft_consensus.cc:2781] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:09:48.279932 18003 raft_consensus.cc:482] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:09:48.280052 18003 raft_consensus.cc:3037] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:09:48.282182 18003 raft_consensus.cc:504] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43191 } } peers { permanent_uuid: "1954a3594b794a968dbbc2ac1e788c06" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42623 } } peers { permanent_uuid: "a4b985106a5a4a1eb18d158165741c48" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44207 } }
I20260924 10:09:48.282809 18003 leader_election.cc:288] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [CANDIDATE]: Term 1 election: Requested vote from peers 1954a3594b794a968dbbc2ac1e788c06 (127.6.2.195:42623), a4b985106a5a4a1eb18d158165741c48 (127.6.2.194:44207)
I20260924 10:09:48.283311 17931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a8e5b7686a498db937a80bba2bdc61" candidate_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1954a3594b794a968dbbc2ac1e788c06"
I20260924 10:09:48.283304 17797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a8e5b7686a498db937a80bba2bdc61" candidate_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b985106a5a4a1eb18d158165741c48"
I20260924 10:09:48.283509 17797 raft_consensus.cc:3037] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:09:48.283509 17931 raft_consensus.cc:3037] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:09:48.286626 17797 raft_consensus.cc:2454] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a02d8fd1fa4c498a19cb35be2fd9ef in term 1.
I20260924 10:09:48.287093 17931 raft_consensus.cc:2454] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a02d8fd1fa4c498a19cb35be2fd9ef in term 1.
I20260924 10:09:48.287540 17587 leader_election.cc:302] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a02d8fd1fa4c498a19cb35be2fd9ef, a4b985106a5a4a1eb18d158165741c48; no voters:
I20260924 10:09:48.287899 18003 raft_consensus.cc:2781] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:09:48.288350 18003 raft_consensus.cc:686] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [term 1 LEADER]: Becoming Leader. State: Replica: 31a02d8fd1fa4c498a19cb35be2fd9ef, State: Running, Role: LEADER
I20260924 10:09:48.288849 18003 consensus_queue.cc:237] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43191 } } peers { permanent_uuid: "1954a3594b794a968dbbc2ac1e788c06" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42623 } } peers { permanent_uuid: "a4b985106a5a4a1eb18d158165741c48" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44207 } }
I20260924 10:09:48.293150 17486 catalog_manager.cc:5495] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 31a02d8fd1fa4c498a19cb35be2fd9ef (127.6.2.193). New cstate: current_term: 1 leader_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31a02d8fd1fa4c498a19cb35be2fd9ef" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1954a3594b794a968dbbc2ac1e788c06" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b985106a5a4a1eb18d158165741c48" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44207 } health_report { overall_health: UNKNOWN } } }
I20260924 10:09:48.371712 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:48.365998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.6.2.193:43191)
0924 10:09:48.366202 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:48.366207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:48.366223 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:09:48.366312 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:48.366315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:48.366399 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:48.366526 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:09:48.366881 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.367649 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.369444 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.369815 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.369949 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:48.370425 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:09:48.371149 (+ 724us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:09:48.371255 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:48.371364 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 10:09:48.371430 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:09:48.379648 17797 raft_consensus.cc:1261] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48 [term 1 FOLLOWER]: Refusing update from remote peer 31a02d8fd1fa4c498a19cb35be2fd9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:09:48.379779 17931 raft_consensus.cc:1261] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06 [term 1 FOLLOWER]: Refusing update from remote peer 31a02d8fd1fa4c498a19cb35be2fd9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:09:48.380595 18006 consensus_queue.cc:1035] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b985106a5a4a1eb18d158165741c48" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:09:48.380937 18003 consensus_queue.cc:1035] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1954a3594b794a968dbbc2ac1e788c06" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:09:48.401270 18011 mvcc.cc:205] Tried to move back new op lower bound from 7332841833993670656 to 7332841833634652160. Current Snapshot: MvccSnapshot[applied={T|T < 7332841833993670656}]
I20260924 10:09:48.408402 17485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:09:48.408639 17485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:09:48.411995 17485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b7c3e6e1827485a8275b1bf5e67bc9f: Sending DeleteTablet for 3 replicas of tablet f2a8e5b7686a498db937a80bba2bdc61
I20260924 10:09:48.413146 17777 tablet_service.cc:1514] Processing DeleteTablet for tablet f2a8e5b7686a498db937a80bba2bdc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:09:48 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:35640
I20260924 10:09:48.413157 17634 tablet_service.cc:1514] Processing DeleteTablet for tablet f2a8e5b7686a498db937a80bba2bdc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:09:48 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:41704
I20260924 10:09:48.413342 17911 tablet_service.cc:1514] Processing DeleteTablet for tablet f2a8e5b7686a498db937a80bba2bdc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:09:48 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:35206
I20260924 10:09:48.414085 18018 tablet_replica.cc:330] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef: stopping tablet replica
I20260924 10:09:48.414885 18018 raft_consensus.cc:2229] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:09:48.415972 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 17566
I20260924 10:09:48.415972 18018 raft_consensus.cc:2258] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:09:48.416929 18019 tablet_replica.cc:330] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48: stopping tablet replica
I20260924 10:09:48.417373 18019 raft_consensus.cc:2229] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:09:48.417920 18019 raft_consensus.cc:2258] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:09:48.422446 18019 ts_tablet_manager.cc:1905] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:09:48.424916 18018 ts_tablet_manager.cc:1905] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:09:48.431555 18020 tablet_replica.cc:330] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06: stopping tablet replica
I20260924 10:09:48.431983 18019 ts_tablet_manager.cc:1918] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:09:48.432026 18020 raft_consensus.cc:2229] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:09:48.432139 18019 log.cc:1196] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/f2a8e5b7686a498db937a80bba2bdc61
I20260924 10:09:48.432561 18020 raft_consensus.cc:2258] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:09:48.432622 18019 ts_tablet_manager.cc:1939] T f2a8e5b7686a498db937a80bba2bdc61 P a4b985106a5a4a1eb18d158165741c48: Deleting consensus metadata
I20260924 10:09:48.433867 18018 ts_tablet_manager.cc:1918] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:09:48.434070 18018 log.cc:1196] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/f2a8e5b7686a498db937a80bba2bdc61
I20260924 10:09:48.434476 17470 catalog_manager.cc:4841] TS a4b985106a5a4a1eb18d158165741c48 (127.6.2.194:44207): tablet f2a8e5b7686a498db937a80bba2bdc61 (table test-table [id=39a276ce2c8d43fe9f8a0f9b2aa48ae1]) successfully deleted
I20260924 10:09:48.434696 18020 ts_tablet_manager.cc:1905] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:09:48.434958 18018 ts_tablet_manager.cc:1939] T f2a8e5b7686a498db937a80bba2bdc61 P 31a02d8fd1fa4c498a19cb35be2fd9ef: Deleting consensus metadata
I20260924 10:09:48.436616 17473 catalog_manager.cc:4841] TS 31a02d8fd1fa4c498a19cb35be2fd9ef (127.6.2.193:43191): tablet f2a8e5b7686a498db937a80bba2bdc61 (table test-table [id=39a276ce2c8d43fe9f8a0f9b2aa48ae1]) successfully deleted
Sep 24 10:09:48 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:09:48.440918 18020 ts_tablet_manager.cc:1918] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:09:48.441078 18020 log.cc:1196] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/f2a8e5b7686a498db937a80bba2bdc61
I20260924 10:09:48.441573 18020 ts_tablet_manager.cc:1939] T f2a8e5b7686a498db937a80bba2bdc61 P 1954a3594b794a968dbbc2ac1e788c06: Deleting consensus metadata
I20260924 10:09:48.443274 17473 catalog_manager.cc:4841] TS 1954a3594b794a968dbbc2ac1e788c06 (127.6.2.195:42623): tablet f2a8e5b7686a498db937a80bba2bdc61 (table test-table [id=39a276ce2c8d43fe9f8a0f9b2aa48ae1]) successfully deleted
I20260924 10:09:48.446007 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:48.418843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41722 (local address 127.6.2.193:43191)
0924 10:09:48.419028 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:48.419033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:48.419087 (+ 54us) server_negotiation.cc:407] Connection header received
0924 10:09:48.419147 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:48.419152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:48.419222 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:48.419323 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:09:48.422685 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.423872 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.432369 (+ 8497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.433118 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.433271 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:48.438424 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:09:48.438440 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:09:48.438459 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:09:48.438504 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:09:48.442714 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:48.443654 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:48.443660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:48.443663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:48.443865 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:48.444259 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:48.444262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:48.444264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:48.444670 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:09:48.444959 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:48.445370 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 10:09:48.445607 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:09:48.446677 17694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:09:48.545953 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 17566
I20260924 10:09:48.563731 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 17705
Sep 24 10:09:48 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:09:48.577912 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:48.564516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.6.2.194:44207)
0924 10:09:48.564693 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:48.564698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:48.564715 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:09:48.564843 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:48.564847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:48.564908 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:48.564990 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:09:48.566190 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.566883 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.568100 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.568479 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.568567 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:48.572643 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:09:48.572656 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:09:48.572672 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:09:48.572708 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:09:48.575537 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:48.576160 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:48.576166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:48.576169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:48.576306 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:48.576599 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:48.576602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:48.576604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:48.576926 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:09:48.577130 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:48.577367 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 10:09:48.577511 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:09:48.578522 17837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:09:48.668246 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 17705
I20260924 10:09:48.685105 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 17848
Sep 24 10:09:48 dist-test-slave-s3n3 krb5kdc[17010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:09:48.700438 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:09:48.685973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.6.2.195:42623)
0924 10:09:48.686138 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:09:48.686143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:09:48.686160 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:09:48.686212 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:09:48.686215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:09:48.686280 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:09:48.686360 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:09:48.687712 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.688442 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.689768 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:09:48.690155 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:09:48.690250 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:09:48.694551 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:09:48.694566 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:09:48.694583 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:09:48.694621 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:09:48.697475 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:48.698140 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:48.698147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:48.698150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:48.698281 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:09:48.698702 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:09:48.698710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:09:48.698717 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:09:48.699303 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:09:48.699628 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 10:09:48.699903 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 10:09:48.700057 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:09:48.701098 17971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:09:48.790391 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 17848
I20260924 10:09:48.806955 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 17454
I20260924 10:09:48.808142 17516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:09:48.920449 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 17454
I20260924 10:09:48.942731 6155 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 10:09:49.278113 6155 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 10:09:49.278316 6155 mini_ranger.cc:66] Stopping Ranger...
I20260924 10:09:49.625018 6155 mini_ranger.cc:68] Stopped Ranger
2026-09-24 10:09:49.625 UTC [17040] LOG: received smart shutdown request
2026-09-24 10:09:49.629 UTC [17040] LOG: background worker "logical replication launcher" (PID 17049) exited with exit code 1
2026-09-24 10:09:49.629 UTC [17044] LOG: shutting down
2026-09-24 10:09:49.804 UTC [17040] LOG: database system is shut down
2026-09-24T10:09:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54161 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:09:50 dist-test-slave-s3n3 krb5kdc[18034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:09:50 dist-test-slave-s3n3 krb5kdc[18034](info): set up 2 sockets
Sep 24 10:09:50 dist-test-slave-s3n3 krb5kdc[18034](info): commencing operation
krb5kdc: starting...
W20260924 10:09:51.051290 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.009s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:09:51 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:09:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/rangeradmin_127.6.2.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/rangeradmin_127.6.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/rangerlookup_127.6.2.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/rangerlookup_127.6.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/HTTP_127.6.2.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/HTTP_127.6.2.212@KRBTEST.COM.keytab.
I20260924 10:09:51.373705 6155 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskq3UYkq/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/postgres -l logfile start
2026-09-24 10:09:54.142 UTC [18064] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 10:09:54.142 UTC [18064] LOG: listening on IPv4 address "127.6.2.212", port 49489
2026-09-24 10:09:54.146 UTC [18064] LOG: listening on Unix socket "/tmp/.s.PGSQL.49489"
2026-09-24 10:09:54.168 UTC [18067] LOG: database system was shut down at 2026-09-24 10:09:52 UTC
2026-09-24 10:09:54.172 UTC [18064] LOG: database system is ready to accept connections
I20260924 10:09:55.154932 6155 mini_postgres.cc:96] Postgres bound to 49489
2026-09-24 10:09:55.172 UTC [18075] FATAL: database "slave" does not exist
127.6.2.212:49489 - accepting connections
I20260924 10:09:55.172673 6155 mini_ranger.cc:161] Starting Ranger...
I20260924 10:09:55.198693 6155 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 10:09:55.735678 6155 mini_ranger.cc:87] Created ranger Postgres database
I20260924 10:09:55.735805 6155 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-admin
2026-09-24 10:09:56,060 [I] DB FLAVOR :POSTGRES
2026-09-24 10:09:56,060 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:09:56,060 [I] Checking connection..
2026-09-24 10:09:56,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 10:09:56,308 [I] Checking connection passed.
2026-09-24 10:09:56,309 [I] --------- Verifying version history table ---------
2026-09-24 10:09:56,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:09:56,562 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:09:56,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:09:56,819 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:09:56,820 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 10:09:56,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 10:09:57.076 UTC [18139] WARNING: there is no transaction in progress
2026-09-24 10:09:57,083 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 10:09:57,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:09:57,348 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 10:09:57,348 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:09:57,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 10:09:57,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 10:09:57,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s3n3','N') ;"
2026-09-24 10:09:58,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:09:58,425 [I] Table x_portal_user does not exist in database ranger
2026-09-24 10:09:58,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:09:58,695 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 10:09:58,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:09:58,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:09:59,224 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 10:09:59,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 10:10:00.364 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.384 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.404 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.424 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.446 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.469 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.533 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.542 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.560 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.572 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.584 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.601 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.612 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.621 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.630 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:00.639 UTC [18247] WARNING: there is no transaction in progress
2026-09-24 10:10:01,420 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 10:10:01,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 10:10:01,662 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 10:10:01,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:10:01,937 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 10:10:01,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:10:02,211 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 10:10:02,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:10:02,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:10:02,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s3n3';"
2026-09-24 10:10:02,985 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 10:10:02,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 10:10:03,238 [I] DB_PATCHES have already been applied
I20260924 10:10:03.244306 6155 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-admin:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskq3UYkq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 10:10:03.244396 6155 mini_ranger.cc:193] Using host: 127.6.2.212
I20260924 10:10:03.259034 6155 mini_ranger.cc:227] Ranger admin URL: http://127.6.2.212:55299
Sep 24, 2026 10:10:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:10:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:10:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 10:10:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 10:10:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 10:10:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 10:10:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 10:10:04.428730 6155 mini_ranger.cc:160] Time spent starting Ranger: real 9.256s user 0.006s sys 0.035s
Sep 24, 2026 10:10:24 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 10:10:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 10:10:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 10:10:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 10:10:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 10:10:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 10:10:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 10:10:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 10:10:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 10:10:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 10:10:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 10:10:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 10:10:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 10:10:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 10:10:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 10:10:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 10:10:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 10:10:25 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 10:10:26.310055 6155 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.2.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.2.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/rangerkms_127.6.2.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.2.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/rangerkms_127.6.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.2.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.2.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.2.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/HTTP_127.6.2.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.2.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/HTTP_127.6.2.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 10:10:26 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244626, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 10:10:26.507326 6155 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 10:10:26.531379 6155 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 10:10:27.606123 6155 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 10:10:27.606247 6155 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-kms
I20260924 10:10:27.606309 6155 mini_ranger_kms.cc:227] Using postgres at 127.6.2.212:49489
2026-09-24 10:10:27,957 [I] DB FLAVOR :POSTGRES
2026-09-24 10:10:27,957 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:10:27,957 [I] Checking connection
2026-09-24 10:10:27,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 10:10:28,242 [I] connection success
2026-09-24 10:10:28,242 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 10:10:28,242 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 10:10:28,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 10:10:28,492 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 10:10:28,493 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:10:28,493 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 10:10:28,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.2.212:49489/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 10:10:28,768 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 10:10:29.242280 6155 mini_ranger_kms.cc:310] Created kms service
I20260924 10:10:29.415019 6155 mini_ranger_kms.cc:326] Created kudu user
I20260924 10:10:29.464356 6155 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 10:10:29 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244626, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:10:29.863260 6155 mini_ranger_kms.cc:384] Added ranger policy
I20260924 10:10:29.863571 6155 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-kms:/tmp/dist-test-taskq3UYkq/build/asan/bin/postgresql.jar:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskq3UYkq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 10:10:29.863615 6155 mini_ranger_kms.cc:240] Using host: 127.6.2.212
I20260924 10:10:29.881436 6155 mini_ranger_kms.cc:276] Ranger KMS PID: 18428
I20260924 10:10:29.881569 6155 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.2.212:55211
Sep 24, 2026 10:10:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:10:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:10:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 10:10:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 10:10:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 10:10:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 10:10:32.235030 6155 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.728s user 0.056s sys 0.318s
I20260924 10:10:32.235312 6155 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 10:10:32.235486 6155 mini_ranger_kms.cc:401] 127.6.2.212:55211/kms/v1/keys
Sep 24 10:10:32 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790244632, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.2.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 10:10:32 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244632, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.2.212@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
Sep 24 10:10:33 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244632, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.2.212@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
Sep 24 10:10:35 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:10:35.496562 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:38343
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:38343
--ranger_config_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.2.212:55211/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:10:35.830627 18478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:35.830983 18478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:35.831163 18478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:35.838609 18478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:10:35.838771 18478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:35.838883 18478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:35.838986 18478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:10:35.839092 18478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:10:35.849763 18478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.2.212:55211/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:38343
--ranger_config_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:38343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:35.851282 18478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:35.852649 18478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:35.865615 18483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:35.865666 18486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:35.865577 18484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:35.868666 18478 server_base.cc:1034] running on GCE node
I20260924 10:10:35.869606 18478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:35.871250 18478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:35.872514 18478 hybrid_clock.cc:648] HybridClock initialized: now 1790244635872467 us; error 35 us; skew 500 ppm
I20260924 10:10:35.872884 18478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:35 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244635, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:35.882257 18478 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:10:35.884085 18478 webserver.cc:460] Webserver started at http://127.6.2.254:42507/ using document root <none> and password file <none>
I20260924 10:10:35.884820 18478 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:35.884959 18478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:35.885288 18478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:10:35 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244635, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:10:35.936640 18478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "f7cecfc40a96472c82520c04835ba294"
format_stamp: "Formatted at 2026-09-24 10:10:35 on dist-test-slave-s3n3"
server_key: "e755faba6e492202dcf91894fe99a6b9"
server_key_iv: "b6a27c7107478228a58d74dcc9886ebb"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:35.937538 18478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "f7cecfc40a96472c82520c04835ba294"
format_stamp: "Formatted at 2026-09-24 10:10:35 on dist-test-slave-s3n3"
server_key: "e755faba6e492202dcf91894fe99a6b9"
server_key_iv: "b6a27c7107478228a58d74dcc9886ebb"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:35.943275 18478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:10:35.960153 18493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:35.961647 18478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:10:35.961864 18478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "f7cecfc40a96472c82520c04835ba294"
format_stamp: "Formatted at 2026-09-24 10:10:35 on dist-test-slave-s3n3"
server_key: "e755faba6e492202dcf91894fe99a6b9"
server_key_iv: "b6a27c7107478228a58d74dcc9886ebb"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:35.962105 18478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:35.989264 18478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:35.993431 18478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:35.993753 18478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:36.013038 18478 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:38343
I20260924 10:10:36.013129 18544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:38343 every 8 connection(s)
I20260924 10:10:36.014510 18478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:10:36.018570 18545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:36.024149 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 18478
I20260924 10:10:36.024415 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:10:36.025134 6155 external_mini_cluster.cc:1428] Setting key cd7fd09044630828f6d332bed4b38c93
I20260924 10:10:36.030874 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294: Bootstrap starting.
I20260924 10:10:36.034888 18545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:36.036008 18545 log.cc:824] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:36.039312 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294: No bootstrap required, opened a new log
Sep 24 10:10:36 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:36.050108 18545 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cecfc40a96472c82520c04835ba294" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38343 } }
I20260924 10:10:36.050580 18545 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:36.050691 18545 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cecfc40a96472c82520c04835ba294, State: Initialized, Role: FOLLOWER
I20260924 10:10:36.050887 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:36.027387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.6.2.254:38343)
0924 10:10:36.028249 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:36.028266 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:36.028317 (+ 51us) server_negotiation.cc:407] Connection header received
0924 10:10:36.030638 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:36.030668 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:36.031230 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:36.031664 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:36.032917 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:36.034879 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:36.036297 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:36.037509 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:36.037733 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:36.041644 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:36.041655 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:36.041676 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:36.041726 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:36.046130 (+ 4404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:36.046867 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:36.046876 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:36.046885 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:36.047097 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:36.047476 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:36.047481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:36.047485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:36.048286 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:36.048614 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:36.049050 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 10:10:36.049494 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":337,"threads_started":1}
I20260924 10:10:36.051469 18545 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cecfc40a96472c82520c04835ba294" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38343 } }
I20260924 10:10:36.051733 18545 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:10:36.051894 18545 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:10:36.052090 18545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:36.054988 18545 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cecfc40a96472c82520c04835ba294" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38343 } }
I20260924 10:10:36.055766 18545 leader_election.cc:302] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7cecfc40a96472c82520c04835ba294; no voters:
I20260924 10:10:36.056468 18545 leader_election.cc:288] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:10:36.056588 18550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:10:36.057242 18550 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 1 LEADER]: Becoming Leader. State: Replica: f7cecfc40a96472c82520c04835ba294, State: Running, Role: LEADER
I20260924 10:10:36.057897 18550 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cecfc40a96472c82520c04835ba294" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38343 } }
I20260924 10:10:36.058805 18545 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:10:36.061861 18551 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7cecfc40a96472c82520c04835ba294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cecfc40a96472c82520c04835ba294" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38343 } } }
I20260924 10:10:36.062245 18551 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:36.061194 18552 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7cecfc40a96472c82520c04835ba294. Latest consensus state: current_term: 1 leader_uuid: "f7cecfc40a96472c82520c04835ba294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cecfc40a96472c82520c04835ba294" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38343 } } }
I20260924 10:10:36.064715 18552 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:36.068446 18558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:10:36.068958 18545 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 10:10:36.081413 18558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:10:36.104024 18558 catalog_manager.cc:1347] Generated new cluster ID: 8d06965e70a54d859002a70072c8fb7b
I20260924 10:10:36.104218 18558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:10:36.132750 18558 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:10:36.134914 18558 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:10:36.143467 18558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294: Generated new TSK 0
I20260924 10:10:36.144544 18558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:10:36.887387 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.2.212:55211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38343
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:10:37 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790244637, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 10:10:37.284108 18590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:37.284436 18590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:37.284605 18590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:37.292332 18590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:37.292490 18590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:37.292686 18590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:10:37.305130 18590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.2.212:55211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:37.306874 18590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:37.308238 18590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:37.325433 18600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:37.326745 18601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:37.335716 18590 server_base.cc:1034] running on GCE node
W20260924 10:10:37.336529 18603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:37.337378 18590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:37.338766 18590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:37.340075 18590 hybrid_clock.cc:648] HybridClock initialized: now 1790244637339995 us; error 71 us; skew 500 ppm
I20260924 10:10:37.340490 18590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:37 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244637, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:37.344992 18590 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:10:37.351334 18590 webserver.cc:460] Webserver started at http://127.6.2.193:40873/ using document root <none> and password file <none>
I20260924 10:10:37.352319 18590 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:37.352463 18590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:37.352833 18590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:10:37 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244637, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:10:37.375845 18590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "68ee23ff3e48487fb01463de9f3170a3"
format_stamp: "Formatted at 2026-09-24 10:10:37 on dist-test-slave-s3n3"
server_key: "278cef5303c1bb5d34fb26f19d9ea93e"
server_key_iv: "3802991ed44ddbc51371416717ab8596"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:37.377058 18590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "68ee23ff3e48487fb01463de9f3170a3"
format_stamp: "Formatted at 2026-09-24 10:10:37 on dist-test-slave-s3n3"
server_key: "278cef5303c1bb5d34fb26f19d9ea93e"
server_key_iv: "3802991ed44ddbc51371416717ab8596"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:37.384649 18590 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260924 10:10:37.416572 18610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:37.418570 18590 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.002s sys 0.000s
I20260924 10:10:37.418790 18590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "68ee23ff3e48487fb01463de9f3170a3"
format_stamp: "Formatted at 2026-09-24 10:10:37 on dist-test-slave-s3n3"
server_key: "278cef5303c1bb5d34fb26f19d9ea93e"
server_key_iv: "3802991ed44ddbc51371416717ab8596"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:37.419044 18590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:37.444530 18590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:37.449774 18590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:37.450105 18590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:37.451347 18590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:10:37.453197 18590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:10:37.453301 18590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:37.453469 18590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:10:37.453558 18590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:37.553385 18590 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:40209
I20260924 10:10:37.553432 18722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:40209 every 8 connection(s)
I20260924 10:10:37.555200 18590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:10:37.559361 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 18590
I20260924 10:10:37.559571 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:10:37.560484 6155 external_mini_cluster.cc:1428] Setting key 0da6c57929eb91771ed10cdbb7b48314
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
Sep 24 10:10:37 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244637, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:37.617755 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:37.559466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:60473 (local address 127.6.2.254:38343)
0924 10:10:37.596178 (+ 36712us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:37.596185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:37.596214 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:10:37.596702 (+ 488us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x60c000061c40. stack: 00007fc54b53c66e 00007fc54b53c430 00007fc54a5c35ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fc55e423775 00007fc55643a270 00007fc54b560358 00000000009da5bb 00007fc54b57ffd7 00007fc54b57bbea 00007fc54b580cee 00007fc551fefe4a 00007fc551fef377 00007fc55200032d
0924 10:10:37.597069 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:37.597074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:37.597173 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:37.597268 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:37.599716 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:37.600904 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:37.603443 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:37.604114 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:37.604226 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:37.608950 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:37.608957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:37.608962 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:37.609012 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:37.612742 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:37.613774 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:37.613780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:37.613784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:37.613942 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:37.614511 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:37.614516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:37.614520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:37.614879 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:37.615027 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:37.617055 (+ 2028us) server_negotiation.cc:299] Negotiation successful
0924 10:10:37.617308 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30466,"spinlock_wait_cycles":2547968,"thread_start_us":35602,"threads_started":1}
I20260924 10:10:37.619087 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:37.558885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:38343 (local address 127.6.2.193:60473)
0924 10:10:37.592365 (+ 33480us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:37.592433 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:37.594801 (+ 2368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:37.597721 (+ 2920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:37.597733 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:37.598621 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:37.599499 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:37.599524 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:37.601122 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:37.601127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:37.603262 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:37.603279 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:37.604596 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:37.604604 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:37.604913 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:37.605720 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:37.605750 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:37.608677 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:37.612941 (+ 4264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:37.612951 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:37.612971 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:37.613608 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:37.614086 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:37.614091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:37.614094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:37.614366 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:37.615124 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:37.615133 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:37.615607 (+ 474us) client_negotiation.cc:769] Sending connection context
0924 10:10:37.615933 (+ 326us) client_negotiation.cc:240] Negotiation successful
0924 10:10:37.616510 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33099,"thread_start_us":35497,"threads_started":1}
I20260924 10:10:37.621062 18723 heartbeater.cc:347] Connected to a master server at 127.6.2.254:38343
I20260924 10:10:37.621423 18723 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:37.622314 18723 heartbeater.cc:511] Master 127.6.2.254:38343 requested a full tablet report, sending...
I20260924 10:10:37.624812 18510 ts_manager.cc:194] Registered new tserver with Master: 68ee23ff3e48487fb01463de9f3170a3 (127.6.2.193:40209)
I20260924 10:10:37.626729 18510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:60473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:10:37.643906 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.2.212:55211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38343
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:10:37 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244637, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
Sep 24 10:10:37 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244637, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
W20260924 10:10:38.119342 18729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:38.119809 18729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:38.119992 18729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:38.131798 18729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:38.132027 18729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:38.132251 18729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:10:38.148021 18729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.2.212:55211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:38.149989 18729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:38.151861 18729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:38.188295 18736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:38.192945 18739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:38.193121 18737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:38.195102 18729 server_base.cc:1034] running on GCE node
I20260924 10:10:38.196036 18729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:38.197212 18729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:38.198558 18729 hybrid_clock.cc:648] HybridClock initialized: now 1790244638198457 us; error 83 us; skew 500 ppm
I20260924 10:10:38.198936 18729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:38 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244638, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:38.202334 18729 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:10:38.204334 18729 webserver.cc:460] Webserver started at http://127.6.2.194:33793/ using document root <none> and password file <none>
I20260924 10:10:38.205092 18729 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:38.205240 18729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:38.205605 18729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:10:38 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244638, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:10:38.241101 18729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "e48c6c0f641f4c1bbf7e9da9694d5070"
format_stamp: "Formatted at 2026-09-24 10:10:38 on dist-test-slave-s3n3"
server_key: "6dd8ec22e06cc7580112f73439ca1128"
server_key_iv: "4a6cbfa1ca9664517bcef04357d5d075"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:38.242261 18729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "e48c6c0f641f4c1bbf7e9da9694d5070"
format_stamp: "Formatted at 2026-09-24 10:10:38 on dist-test-slave-s3n3"
server_key: "6dd8ec22e06cc7580112f73439ca1128"
server_key_iv: "4a6cbfa1ca9664517bcef04357d5d075"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:38.253577 18729 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.009s sys 0.000s
I20260924 10:10:38.274073 18753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:38.276048 18729 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:10:38.276255 18729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "e48c6c0f641f4c1bbf7e9da9694d5070"
format_stamp: "Formatted at 2026-09-24 10:10:38 on dist-test-slave-s3n3"
server_key: "6dd8ec22e06cc7580112f73439ca1128"
server_key_iv: "4a6cbfa1ca9664517bcef04357d5d075"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:38.276530 18729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:38.304860 18729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:38.310274 18729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:38.310616 18729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:38.311959 18729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:10:38.313704 18729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:10:38.313803 18729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:38.313912 18729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:10:38.313959 18729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:38.355527 18729 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:41215
I20260924 10:10:38.355705 18865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:41215 every 8 connection(s)
I20260924 10:10:38.357272 18729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:10:38.359509 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 18729
I20260924 10:10:38.359728 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:10:38.360219 6155 external_mini_cluster.cc:1428] Setting key 47f2c608ca46ed722b38dd1e13e03b02
Sep 24 10:10:38 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244638, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:10:38.389452 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:38.362773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:50519 (local address 127.6.2.254:38343)
0924 10:10:38.369370 (+ 6597us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:38.369380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:38.369407 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:10:38.369484 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:38.369493 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:38.369597 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:38.369739 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:38.371826 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:38.372910 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:38.375472 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:38.376331 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:38.376504 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:38.381413 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:38.381420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:38.381425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:38.381469 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:38.385129 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:38.386333 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:38.386339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:38.386343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:38.386495 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:38.387106 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:38.387111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:38.387114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:38.387494 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:38.387646 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:38.388788 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0924 10:10:38.389061 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6407,"thread_start_us":6267,"threads_started":1}
I20260924 10:10:38.390616 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:38.363469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:38343 (local address 127.6.2.194:50519)
0924 10:10:38.364409 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:38.364469 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:38.367530 (+ 3061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:38.370060 (+ 2530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:38.370074 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:38.370728 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:38.371575 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:38.371602 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:38.373109 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:38.373115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:38.375289 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:38.375306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:38.376611 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:38.376618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:38.376991 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:38.377898 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:38.377930 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:38.381101 (+ 3171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:38.385337 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:38.385352 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:38.385379 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:38.386162 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:38.386631 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:38.386637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:38.386641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:38.386924 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:38.387657 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:38.387668 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:38.388177 (+ 509us) client_negotiation.cc:769] Sending connection context
0924 10:10:38.388579 (+ 402us) client_negotiation.cc:240] Negotiation successful
0924 10:10:38.389230 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":4442,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
I20260924 10:10:38.392897 18866 heartbeater.cc:347] Connected to a master server at 127.6.2.254:38343
I20260924 10:10:38.393218 18866 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:38.394048 18866 heartbeater.cc:511] Master 127.6.2.254:38343 requested a full tablet report, sending...
I20260924 10:10:38.396481 18510 ts_manager.cc:194] Registered new tserver with Master: e48c6c0f641f4c1bbf7e9da9694d5070 (127.6.2.194:41215)
I20260924 10:10:38.397719 18510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:50519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:10:38.424538 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.2.212:55211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38343
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:10:38.630621 18723 heartbeater.cc:503] Master 127.6.2.254:38343 was elected leader, sending a full tablet report...
W20260924 10:10:38.769788 18872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:38.770118 18872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:38.770282 18872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:38.777921 18872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:38.778079 18872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:38.778268 18872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:10:38.788316 18872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.2.212:55211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:38343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:38.789667 18872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:38.791003 18872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:38.804673 18880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:38.804749 18878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:38.804728 18877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:38.806128 18872 server_base.cc:1034] running on GCE node
I20260924 10:10:38.806792 18872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:38.808271 18872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:38.809531 18872 hybrid_clock.cc:648] HybridClock initialized: now 1790244638809449 us; error 56 us; skew 500 ppm
I20260924 10:10:38.809890 18872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:38 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244638, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:38.813288 18872 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:10:38.815013 18872 webserver.cc:460] Webserver started at http://127.6.2.195:36537/ using document root <none> and password file <none>
I20260924 10:10:38.815799 18872 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:38.815963 18872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:38.816296 18872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:10:38 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244638, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:10:38.835716 18872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "5d9dad7486054aa99fa052291e222d90"
format_stamp: "Formatted at 2026-09-24 10:10:38 on dist-test-slave-s3n3"
server_key: "e45d4022d10594e53481ab41e9df6e87"
server_key_iv: "994b15e98055f5aa8be87033b2a01429"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:38.837006 18872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "5d9dad7486054aa99fa052291e222d90"
format_stamp: "Formatted at 2026-09-24 10:10:38 on dist-test-slave-s3n3"
server_key: "e45d4022d10594e53481ab41e9df6e87"
server_key_iv: "994b15e98055f5aa8be87033b2a01429"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:38.844163 18872 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 10:10:38.862293 18887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:38.864091 18872 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 10:10:38.864276 18872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "5d9dad7486054aa99fa052291e222d90"
format_stamp: "Formatted at 2026-09-24 10:10:38 on dist-test-slave-s3n3"
server_key: "e45d4022d10594e53481ab41e9df6e87"
server_key_iv: "994b15e98055f5aa8be87033b2a01429"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:38.864530 18872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:38.902031 18872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:38.906356 18872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:38.906638 18872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:38.907678 18872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:10:38.909013 18872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:10:38.909089 18872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:38.909245 18872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:10:38.909298 18872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:38.945269 18872 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:36481
I20260924 10:10:38.945312 18999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:36481 every 8 connection(s)
I20260924 10:10:38.946686 18872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:10:38.950122 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 18872
I20260924 10:10:38.950304 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:10:38.950861 6155 external_mini_cluster.cc:1428] Setting key ce776a08fb2fbecf1eab816bc3f544ad
Sep 24 10:10:38 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244638, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:38.970495 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:38.949993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:45543 (local address 127.6.2.254:38343)
0924 10:10:38.950540 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:38.950545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:38.951900 (+ 1355us) server_negotiation.cc:407] Connection header received
0924 10:10:38.954492 (+ 2592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:38.954498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:38.954597 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:38.954727 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:38.956292 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:38.957072 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:38.958779 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:38.959352 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:38.959505 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:38.963099 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:38.963109 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:38.963115 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:38.963163 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:38.966322 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:38.967454 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:38.967458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:38.967461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:38.967592 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:38.968035 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:38.968038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:38.968040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:38.968326 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:38.968449 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:38.969486 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0924 10:10:38.969848 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":273,"threads_started":1}
I20260924 10:10:38.970988 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:38.950644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:38343 (local address 127.6.2.195:45543)
0924 10:10:38.951706 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:38.951779 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:38.954189 (+ 2410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:38.954985 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:38.954994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:38.955514 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:38.956116 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:38.956135 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:38.957246 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:38.957256 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:38.958619 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:38.958632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:38.959594 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:38.959602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:38.959893 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:38.960607 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:38.960631 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:38.962798 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:38.966579 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:38.966589 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:38.966609 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:38.967344 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:38.967692 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:38.967729 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:38.967731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:38.967930 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:38.968466 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:38.968472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:38.968819 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 10:10:38.969097 (+ 278us) client_negotiation.cc:240] Negotiation successful
0924 10:10:38.969570 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":473,"threads_started":1}
I20260924 10:10:38.972975 19000 heartbeater.cc:347] Connected to a master server at 127.6.2.254:38343
I20260924 10:10:38.973299 19000 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:38.973999 19000 heartbeater.cc:511] Master 127.6.2.254:38343 requested a full tablet report, sending...
I20260924 10:10:38.975549 18510 ts_manager.cc:194] Registered new tserver with Master: 5d9dad7486054aa99fa052291e222d90 (127.6.2.195:36481)
I20260924 10:10:38.976395 18510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:45543
I20260924 10:10:38.980625 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:10:38.982550 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 18590
Sep 24 10:10:38 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:10:39.004201 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:38.983939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.6.2.193:40209)
0924 10:10:38.984909 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:38.984922 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:38.984970 (+ 48us) server_negotiation.cc:407] Connection header received
0924 10:10:38.985274 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:38.985293 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:38.985860 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:38.986148 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:38.987900 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:38.988985 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:38.991381 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:38.992020 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:38.992181 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:38.997516 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:38.997528 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:38.997544 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:38.997579 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:39.001172 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:39.001969 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:39.001982 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:39.001985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:39.002126 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:39.002535 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:39.002541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:39.002549 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:39.002880 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:39.003225 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:39.003561 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 10:10:39.003841 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":486,"threads_started":1}
I20260924 10:10:39.005910 18718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:39.096864 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 18590
I20260924 10:10:39.113741 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 18729
Sep 24 10:10:39 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:10:39.132444 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:39.115018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.6.2.194:41215)
0924 10:10:39.115688 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:39.115695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:39.115726 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:10:39.115794 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:39.115801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:39.116039 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:39.116183 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:39.118101 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:39.119275 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:39.120635 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:39.121220 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:39.121336 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:39.126108 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:39.126122 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:39.126138 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:39.126177 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:39.129190 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:39.130301 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:39.130308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:39.130310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:39.130475 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:39.130830 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:39.130833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:39.130835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:39.131262 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:39.131534 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:39.131794 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 10:10:39.132012 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":348,"threads_started":1}
I20260924 10:10:39.133765 18861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:39.222321 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 18729
I20260924 10:10:39.242441 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 18872
Sep 24 10:10:39 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:10:39.261876 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:39.243587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.6.2.195:36481)
0924 10:10:39.244218 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:39.244225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:39.244267 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:10:39.244329 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:39.244335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:39.244553 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:39.244679 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:39.246112 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:39.247405 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:39.248725 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:39.249366 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:39.249476 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:39.255022 (+ 5546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:39.255036 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:39.255053 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:39.255329 (+ 276us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:39.258802 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:39.259701 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:39.259710 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:39.259713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:39.259888 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:39.260301 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:39.260304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:39.260306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:39.260667 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:39.260909 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:39.261241 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 10:10:39.261464 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":338,"threads_started":1}
I20260924 10:10:39.263327 18995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:39.351720 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 18872
I20260924 10:10:39.368508 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 18478
I20260924 10:10:39.369809 18540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:39.479709 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 18478
I20260924 10:10:39.503742 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:38343
--webserver_interface=127.6.2.254
--webserver_port=42507
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:38343
--ranger_config_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.2.212:55211/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.2.212:55211/kms with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:10:39.836525 19012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:39.836826 19012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:39.836977 19012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:39.845233 19012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:10:39.845352 19012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:39.845448 19012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:39.845554 19012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:10:39.845650 19012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:10:39.855942 19012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.2.212:55211/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:38343
--ranger_config_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:38343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=42507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:39.857278 19012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:39.858635 19012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:39.871579 19018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:39.871542 19020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:39.871621 19017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:39.872174 19012 server_base.cc:1034] running on GCE node
I20260924 10:10:39.872968 19012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:39.874456 19012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:39.875715 19012 hybrid_clock.cc:648] HybridClock initialized: now 1790244639875648 us; error 66 us; skew 500 ppm
I20260924 10:10:39.876071 19012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:39 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244639, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:39.879210 19012 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:10:39.880756 19012 webserver.cc:460] Webserver started at http://127.6.2.254:42507/ using document root <none> and password file <none>
I20260924 10:10:39.881422 19012 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:39.881567 19012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:10:39 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244639, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:10:39.950158 19012 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 10:10:39.953289 19028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:39.954649 19012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:10:39.954854 19012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "f7cecfc40a96472c82520c04835ba294"
format_stamp: "Formatted at 2026-09-24 10:10:35 on dist-test-slave-s3n3"
server_key: "e755faba6e492202dcf91894fe99a6b9"
server_key_iv: "b6a27c7107478228a58d74dcc9886ebb"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:39.955910 19012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:39.983220 19012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:39.987329 19012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:39.987661 19012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:40.008513 19012 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:38343
I20260924 10:10:40.008530 19079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:38343 every 8 connection(s)
I20260924 10:10:40.010097 19012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:10:40.011523 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 19012
I20260924 10:10:40.013005 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.2.212:55211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:40209
--local_ip_for_outbound_sockets=127.6.2.193
--tserver_master_addrs=127.6.2.254:38343
--webserver_port=40873
--webserver_interface=127.6.2.193
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.2.212:55211/kms with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:10:40.018010 19080 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:10:40.021826 19080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294: Bootstrap starting.
I20260924 10:10:40.049635 19080 log.cc:824] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:40.059933 19080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:10:40.060686 19080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294: Bootstrap complete.
I20260924 10:10:40.070081 19080 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cecfc40a96472c82520c04835ba294" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38343 } }
I20260924 10:10:40.070984 19080 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cecfc40a96472c82520c04835ba294, State: Initialized, Role: FOLLOWER
I20260924 10:10:40.071683 19080 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cecfc40a96472c82520c04835ba294" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38343 } }
I20260924 10:10:40.071930 19080 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:10:40.072067 19080 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:10:40.072238 19080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:10:40.076203 19080 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cecfc40a96472c82520c04835ba294" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38343 } }
I20260924 10:10:40.076687 19080 leader_election.cc:302] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7cecfc40a96472c82520c04835ba294; no voters:
I20260924 10:10:40.077263 19080 leader_election.cc:288] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:10:40.077455 19084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:10:40.078408 19084 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [term 2 LEADER]: Becoming Leader. State: Replica: f7cecfc40a96472c82520c04835ba294, State: Running, Role: LEADER
I20260924 10:10:40.079030 19084 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cecfc40a96472c82520c04835ba294" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38343 } }
I20260924 10:10:40.079258 19080 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:10:40.082110 19086 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7cecfc40a96472c82520c04835ba294. Latest consensus state: current_term: 2 leader_uuid: "f7cecfc40a96472c82520c04835ba294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cecfc40a96472c82520c04835ba294" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38343 } } }
I20260924 10:10:40.082940 19086 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:40.082543 19085 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f7cecfc40a96472c82520c04835ba294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cecfc40a96472c82520c04835ba294" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 38343 } } }
I20260924 10:10:40.083711 19085 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294 [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:40.094802 19091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:10:40.100970 19091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:10:40.102422 19091 catalog_manager.cc:1259] Loaded cluster ID: 8d06965e70a54d859002a70072c8fb7b
I20260924 10:10:40.102581 19091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:10:40.106658 19091 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:10:40.108203 19091 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294: Loaded TSK: 0
I20260924 10:10:40.110059 19091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:10:40.110610 19080 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 10:10:40.481859 19082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:40.482390 19082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:40.482596 19082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:40.494689 19082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:40.494957 19082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:40.495208 19082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:10:40.511335 19082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.2.212:55211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:40209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=40873
--tserver_master_addrs=127.6.2.254:38343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:40.513315 19082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:40.515100 19082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:40.535851 19118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:40.535917 19117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:40.543392 19120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:40.544416 19082 server_base.cc:1034] running on GCE node
I20260924 10:10:40.545521 19082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:40.550417 19082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:40.553977 19082 hybrid_clock.cc:648] HybridClock initialized: now 1790244640553074 us; error 889 us; skew 500 ppm
I20260924 10:10:40.554481 19082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:40 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244640, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:40.559181 19082 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:10:40.561404 19082 webserver.cc:460] Webserver started at http://127.6.2.193:40873/ using document root <none> and password file <none>
I20260924 10:10:40.562448 19082 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:40.562623 19082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:10:40 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244640, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:10:40.593585 19082 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 10:10:40.597689 19128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:40.599587 19082 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:10:40.599892 19082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "68ee23ff3e48487fb01463de9f3170a3"
format_stamp: "Formatted at 2026-09-24 10:10:37 on dist-test-slave-s3n3"
server_key: "278cef5303c1bb5d34fb26f19d9ea93e"
server_key_iv: "3802991ed44ddbc51371416717ab8596"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:40.601410 19082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:40.626492 19082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:40.632026 19082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:40.632411 19082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:40.635147 19082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:10:40.639961 19082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:10:40.640151 19082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 10:10:40.640365 19082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:10:40.640522 19082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:40.698247 19082 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:40209
I20260924 10:10:40.698326 19241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:40209 every 8 connection(s)
I20260924 10:10:40.700196 19082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:10:40.710019 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 19082
I20260924 10:10:40.710839 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.2.212:55211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:41215
--local_ip_for_outbound_sockets=127.6.2.194
--tserver_master_addrs=127.6.2.254:38343
--webserver_port=33793
--webserver_interface=127.6.2.194
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.2.212:55211/kms with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:10:40 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244640, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:40.741032 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:40.703893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:38343 (local address 127.6.2.193:48001)
0924 10:10:40.709891 (+ 5998us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:40.709956 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:40.712988 (+ 3032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:40.715608 (+ 2620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:40.715621 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:40.716304 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:40.717102 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:40.717129 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:40.720135 (+ 3006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:40.720139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:40.723521 (+ 3382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:40.723538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:40.726784 (+ 3246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:40.726797 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:40.727161 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:40.728012 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:40.728048 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:40.731798 (+ 3750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:40.736416 (+ 4618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:40.736426 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:40.736442 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:40.736893 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:40.737383 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:40.737388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:40.737392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:40.737593 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:40.738452 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:40.738461 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:40.738783 (+ 322us) client_negotiation.cc:769] Sending connection context
0924 10:10:40.739045 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 10:10:40.739653 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5327,"thread_start_us":14880,"threads_started":1}
I20260924 10:10:40.741019 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:40.704969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:48001 (local address 127.6.2.254:38343)
0924 10:10:40.708927 (+ 3958us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:40.708947 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:40.711136 (+ 2189us) server_negotiation.cc:407] Connection header received
0924 10:10:40.713785 (+ 2649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:40.713810 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:40.714341 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:40.714725 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:40.717963 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:40.719911 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:40.723723 (+ 3812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:40.726451 (+ 2728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:40.726673 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:40.732144 (+ 5471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:40.732156 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:40.732178 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:40.732250 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:40.736246 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:40.737053 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:40.737069 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:40.737074 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:40.737274 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:40.737728 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:40.737733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:40.737737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:40.738298 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:40.738508 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:40.739312 (+ 804us) server_negotiation.cc:299] Negotiation successful
0924 10:10:40.739683 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3636,"thread_start_us":3594,"threads_started":1}
I20260924 10:10:40.745204 19242 heartbeater.cc:347] Connected to a master server at 127.6.2.254:38343
I20260924 10:10:40.745545 19242 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:40.746503 19242 heartbeater.cc:511] Master 127.6.2.254:38343 requested a full tablet report, sending...
W20260924 10:10:40.749459 19242 heartbeater.cc:644] Failed to heartbeat to 127.6.2.254:38343 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:10:40.749714 19242 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:40.750042 19242 heartbeater.cc:511] Master 127.6.2.254:38343 requested a full tablet report, sending...
I20260924 10:10:40.751299 19242 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:40.751575 19242 heartbeater.cc:511] Master 127.6.2.254:38343 requested a full tablet report, sending...
W20260924 10:10:40.752753 19242 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:10:41.140522 19246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:41.140937 19246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:41.141103 19246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:41.152813 19246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:41.152976 19246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:41.153230 19246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:10:41.164655 19246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.2.212:55211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:41215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=33793
--tserver_master_addrs=127.6.2.254:38343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:41.166026 19246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:41.167346 19246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:41.186461 19260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:41.186444 19261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:41.191795 19263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:41.192679 19246 server_base.cc:1034] running on GCE node
I20260924 10:10:41.193588 19246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:41.195231 19246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:41.196559 19246 hybrid_clock.cc:648] HybridClock initialized: now 1790244641196491 us; error 46 us; skew 500 ppm
I20260924 10:10:41.197023 19246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:41 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244641, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:41.206549 19246 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:10:41.208743 19246 webserver.cc:460] Webserver started at http://127.6.2.194:33793/ using document root <none> and password file <none>
I20260924 10:10:41.209833 19246 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:41.210001 19246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:10:41 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244641, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:10:41.234151 19246 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:10:41.237707 19271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:41.239527 19246 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 10:10:41.239856 19246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "e48c6c0f641f4c1bbf7e9da9694d5070"
format_stamp: "Formatted at 2026-09-24 10:10:38 on dist-test-slave-s3n3"
server_key: "6dd8ec22e06cc7580112f73439ca1128"
server_key_iv: "4a6cbfa1ca9664517bcef04357d5d075"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:41.241364 19246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:41.323927 19246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:41.329519 19246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:41.329835 19246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:41.331100 19246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:10:41.332800 19246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:10:41.332912 19246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:41.333016 19246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:10:41.333091 19246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:10:41 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790244641, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:41.408494 19246 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:41215
I20260924 10:10:41.408653 19385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:41215 every 8 connection(s)
I20260924 10:10:41.410825 19246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:10:41.411974 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 19246
I20260924 10:10:41.412735 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.2.212:55211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:36481
--local_ip_for_outbound_sockets=127.6.2.195
--tserver_master_addrs=127.6.2.254:38343
--webserver_port=36537
--webserver_interface=127.6.2.195
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.2.212:55211/kms with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:10:41 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244641, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:41.446944 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:41.421698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:53979 (local address 127.6.2.254:38343)
0924 10:10:41.422246 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:41.422253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:41.423302 (+ 1049us) server_negotiation.cc:407] Connection header received
0924 10:10:41.426165 (+ 2863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:41.426171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:41.426266 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:41.426421 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:41.428363 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:41.429560 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:41.432120 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:41.432815 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:41.432939 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:41.437732 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:41.437743 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:41.437751 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:41.437802 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:41.442037 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:41.443119 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:41.443128 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:41.443131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:41.443288 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:41.444076 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:41.444082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:41.444087 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:41.444496 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:41.445166 (+ 670us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:41.446313 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0924 10:10:41.446562 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":276,"threads_started":1}
I20260924 10:10:41.448415 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:41.422236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:38343 (local address 127.6.2.194:53979)
0924 10:10:41.423120 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:41.423165 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:41.425859 (+ 2694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:41.426651 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:41.426665 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:41.427354 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:41.428143 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:41.428168 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:41.429732 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:41.429740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:41.431962 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:41.431981 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:41.433060 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:41.433070 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:41.433427 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:41.434287 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:41.434326 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:41.437399 (+ 3073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:41.442207 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:41.442218 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:41.442238 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:41.442921 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:41.443458 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:41.443463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:41.443466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:41.443925 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:41.445253 (+ 1328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:41.445262 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:41.445797 (+ 535us) client_negotiation.cc:769] Sending connection context
0924 10:10:41.446154 (+ 357us) client_negotiation.cc:240] Negotiation successful
0924 10:10:41.446747 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":366,"threads_started":1}
I20260924 10:10:41.450240 19386 heartbeater.cc:347] Connected to a master server at 127.6.2.254:38343
I20260924 10:10:41.450562 19386 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:41.451411 19386 heartbeater.cc:511] Master 127.6.2.254:38343 requested a full tablet report, sending...
I20260924 10:10:41.456156 19045 ts_manager.cc:194] Registered new tserver with Master: e48c6c0f641f4c1bbf7e9da9694d5070 (127.6.2.194:41215)
I20260924 10:10:41.458540 19045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:53979
I20260924 10:10:41.752933 19242 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:41.753417 19242 heartbeater.cc:511] Master 127.6.2.254:38343 requested a full tablet report, sending...
I20260924 10:10:41.754940 19045 ts_manager.cc:194] Registered new tserver with Master: 68ee23ff3e48487fb01463de9f3170a3 (127.6.2.193:40209)
I20260924 10:10:41.756381 19045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:48001
Sep 24 10:10:41 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244641, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
W20260924 10:10:41.862661 19388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:41.863122 19388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:41.863307 19388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:41.876950 19388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:41.877117 19388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:41.877352 19388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:10:41.893643 19388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.2.212:55211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:36481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=36537
--tserver_master_addrs=127.6.2.254:38343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:41.895017 19388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:41.896754 19388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:41.935434 19396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:41.936146 19397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:41.936535 19399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:41.943320 19388 server_base.cc:1034] running on GCE node
I20260924 10:10:41.944041 19388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:41.945257 19388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:41.946489 19388 hybrid_clock.cc:648] HybridClock initialized: now 1790244641946424 us; error 55 us; skew 500 ppm
I20260924 10:10:41.946882 19388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:41 dist-test-slave-s3n3 krb5kdc[18034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244641, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:41.950516 19388 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:10:41.955907 19388 webserver.cc:460] Webserver started at http://127.6.2.195:36537/ using document root <none> and password file <none>
I20260924 10:10:41.956650 19388 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:41.956828 19388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:10:41 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244641, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:10:41.978403 19388 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:10:41.981679 19407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:41.983316 19388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:10:41.983513 19388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "5d9dad7486054aa99fa052291e222d90"
format_stamp: "Formatted at 2026-09-24 10:10:38 on dist-test-slave-s3n3"
server_key: "e45d4022d10594e53481ab41e9df6e87"
server_key_iv: "994b15e98055f5aa8be87033b2a01429"
server_key_version: "kuduclusterkey@0"
I20260924 10:10:41.984543 19388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:42.022492 19388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:42.027845 19388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:42.028156 19388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:42.029294 19388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:10:42.030786 19388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:10:42.030958 19388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:42.031131 19388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:10:42.031213 19388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:42.080646 19388 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:36481
I20260924 10:10:42.080729 19519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:36481 every 8 connection(s)
I20260924 10:10:42.082573 19388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:10:42.093133 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 19388
Sep 24 10:10:42 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244641, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:42.161392 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.086750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:59671 (local address 127.6.2.254:38343)
0924 10:10:42.135547 (+ 48797us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:42.135557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:42.136800 (+ 1243us) server_negotiation.cc:407] Connection header received
0924 10:10:42.139403 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:42.139409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:42.139504 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:42.139608 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:42.141922 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.143385 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.146529 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.147284 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.147382 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.152745 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:42.152752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:42.152756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:42.152827 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:42.156347 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:42.158308 (+ 1961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:42.158314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:42.158318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:42.158465 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:42.159055 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:42.159084 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:42.159087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:42.159483 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:42.159654 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:42.160754 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0924 10:10:42.161001 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48612,"thread_start_us":48497,"threads_started":1}
I20260924 10:10:42.165284 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.086151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:38343 (local address 127.6.2.195:59671)
0924 10:10:42.136211 (+ 50060us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:42.136265 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:42.138884 (+ 2619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:42.140090 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:42.140104 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:42.140749 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:42.141686 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:42.141716 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.143597 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.143603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:42.146301 (+ 2698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:42.146331 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.147764 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.147771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:42.148275 (+ 504us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.149153 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:42.149193 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:42.152416 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:42.157454 (+ 5038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:42.157465 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:42.157486 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:42.158144 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:42.158628 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:42.158633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:42.158636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:42.158912 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:42.159763 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:42.159772 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:42.160224 (+ 452us) client_negotiation.cc:769] Sending connection context
0924 10:10:42.161704 (+ 1480us) client_negotiation.cc:240] Negotiation successful
0924 10:10:42.162316 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49736,"thread_start_us":49516,"threads_started":1}
I20260924 10:10:42.167129 19520 heartbeater.cc:347] Connected to a master server at 127.6.2.254:38343
I20260924 10:10:42.167338 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.094764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.6.2.254:38343)
0924 10:10:42.137196 (+ 42432us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:42.137203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:42.137221 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:10:42.137287 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:42.137291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:42.137398 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:42.139942 (+ 2544us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:42.140129 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.141195 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.143695 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.144347 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.144466 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.151139 (+ 6673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:42.151154 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:42.151161 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:42.151209 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:42.157234 (+ 6025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:42.163198 (+ 5964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:42.163202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:42.163204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:42.163337 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:42.163900 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:42.163905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:42.163908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:42.164276 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:42.164524 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:42.166617 (+ 2093us) server_negotiation.cc:299] Negotiation successful
0924 10:10:42.166905 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42305,"thread_start_us":42759,"threads_started":1}
I20260924 10:10:42.167507 19520 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:42.168450 19520 heartbeater.cc:511] Master 127.6.2.254:38343 requested a full tablet report, sending...
I20260924 10:10:42.170639 19045 ts_manager.cc:194] Registered new tserver with Master: 5d9dad7486054aa99fa052291e222d90 (127.6.2.195:36481)
I20260924 10:10:42.172369 19045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:59671
I20260924 10:10:42.182544 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:10:42 dist-test-slave-s3n3 krb5kdc[18034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244641, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for HTTP/127.6.2.212@KRBTEST.COM
I20260924 10:10:42.226804 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.199929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.6.2.254:38343)
0924 10:10:42.200205 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:42.200214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:42.207102 (+ 6888us) server_negotiation.cc:407] Connection header received
0924 10:10:42.207831 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:42.207837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:42.207947 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:42.208030 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:42.210644 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.211835 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.213794 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.214404 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.214491 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.219329 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:42.219338 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:42.219343 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:42.219391 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:42.223097 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:42.224051 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:42.224056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:42.224059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:42.224215 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:42.224697 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:42.224702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:42.224705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:42.225101 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:42.225333 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:42.226042 (+ 709us) server_negotiation.cc:299] Negotiation successful
0924 10:10:42.226332 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 10:10:42.235802 19045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:10:42.246632 19045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:10:42.328078 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.290072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:40209 (local address 127.0.0.1:41314)
0924 10:10:42.306521 (+ 16449us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:42.306577 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:42.306800 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:42.313583 (+ 6783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:42.313592 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:42.313640 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:42.314137 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:42.314152 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.322073 (+ 7921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.322078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:42.324628 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:42.324641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.326970 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.326977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:42.327245 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.327312 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 10:10:42.327447 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 10:10:42.327671 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16119,"thread_start_us":29256,"threads_started":1}
I20260924 10:10:42.329087 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.290268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.6.2.193:40209)
0924 10:10:42.308508 (+ 18240us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:42.308516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:42.308548 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:10:42.308653 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:42.308661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:42.308951 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:42.309151 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:42.319939 (+ 10788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.321848 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.324837 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.326359 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.326472 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.326715 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:42.328567 (+ 1852us) server_negotiation.cc:299] Negotiation successful
0924 10:10:42.328751 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18072,"thread_start_us":19084,"threads_started":1}
I20260924 10:10:42.333271 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 70d3a5249b4d45e5b1b1823ce653f724 (DEFAULT_TABLE table=test-table [id=1b71eff145af444c9f48903459d6ed5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:10:42.334766 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.300266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:36481 (local address 127.0.0.1:54086)
0924 10:10:42.320372 (+ 20106us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:42.320391 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:42.320493 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:42.322995 (+ 2502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:42.323000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:42.323029 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:42.323495 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:42.323513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.329508 (+ 5995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.329514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:42.331510 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:42.331530 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.333964 (+ 2434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.333971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:42.334157 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.334195 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 10:10:42.334273 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:10:42.334422 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19986,"thread_start_us":19813,"threads_started":1}
I20260924 10:10:42.334805 19176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d3a5249b4d45e5b1b1823ce653f724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:42.336707 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.300525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.6.2.195:36481)
0924 10:10:42.320891 (+ 20366us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:42.320902 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:42.320939 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:10:42.321040 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:42.321049 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:42.322767 (+ 1718us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:42.323014 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:42.323729 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.325347 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.331755 (+ 6408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.333689 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.335169 (+ 1480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.335411 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:42.335492 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 10:10:42.335663 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20149,"thread_start_us":20262,"threads_started":1}
I20260924 10:10:42.340766 19455 tablet_service.cc:1467] Processing CreateTablet for tablet 70d3a5249b4d45e5b1b1823ce653f724 (DEFAULT_TABLE table=test-table [id=1b71eff145af444c9f48903459d6ed5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:10:42.343758 19455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d3a5249b4d45e5b1b1823ce653f724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:42.348084 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.292359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:41215 (local address 127.0.0.1:58848)
0924 10:10:42.319659 (+ 27300us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:42.319682 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:42.319864 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:42.323754 (+ 3890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:42.323771 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:42.323799 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:42.324212 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:42.324226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.326279 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.326285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:42.328549 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:42.328565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.347192 (+ 18627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.347201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:42.347407 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.347448 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 10:10:42.347528 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:10:42.347688 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27134,"thread_start_us":13705,"threads_started":1}
I20260924 10:10:42.349363 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.292922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58848 (local address 127.6.2.194:41215)
0924 10:10:42.307367 (+ 14445us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:42.307376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:42.323157 (+ 15781us) server_negotiation.cc:407] Connection header received
0924 10:10:42.323286 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:42.323294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:42.323545 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:42.324437 (+ 892us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:42.324669 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.326099 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.328773 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.344201 (+ 15428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.348534 (+ 4333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.348768 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:42.348846 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 10:10:42.349035 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14283,"thread_start_us":14165,"threads_started":1}
I20260924 10:10:42.353318 19321 tablet_service.cc:1467] Processing CreateTablet for tablet 70d3a5249b4d45e5b1b1823ce653f724 (DEFAULT_TABLE table=test-table [id=1b71eff145af444c9f48903459d6ed5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:10:42.354801 19321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d3a5249b4d45e5b1b1823ce653f724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:42.371021 19543 tablet_bootstrap.cc:492] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90: Bootstrap starting.
I20260924 10:10:42.372241 19544 tablet_bootstrap.cc:492] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070: Bootstrap starting.
I20260924 10:10:42.376016 19543 tablet_bootstrap.cc:654] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:42.377370 19543 log.cc:824] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:42.377753 19544 tablet_bootstrap.cc:654] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:42.379240 19544 log.cc:824] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:42.382505 19543 tablet_bootstrap.cc:492] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90: No bootstrap required, opened a new log
I20260924 10:10:42.382817 19543 ts_tablet_manager.cc:1397] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260924 10:10:42.398486 19544 tablet_bootstrap.cc:492] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070: No bootstrap required, opened a new log
I20260924 10:10:42.398958 19544 ts_tablet_manager.cc:1397] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.013s
I20260924 10:10:42.400251 19547 tablet_bootstrap.cc:492] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3: Bootstrap starting.
I20260924 10:10:42.402598 19543 raft_consensus.cc:348] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ee23ff3e48487fb01463de9f3170a3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40209 } } peers { permanent_uuid: "e48c6c0f641f4c1bbf7e9da9694d5070" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41215 } } peers { permanent_uuid: "5d9dad7486054aa99fa052291e222d90" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36481 } }
I20260924 10:10:42.403116 19543 raft_consensus.cc:374] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:42.403221 19543 raft_consensus.cc:729] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9dad7486054aa99fa052291e222d90, State: Initialized, Role: FOLLOWER
I20260924 10:10:42.403903 19543 consensus_queue.cc:260] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ee23ff3e48487fb01463de9f3170a3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40209 } } peers { permanent_uuid: "e48c6c0f641f4c1bbf7e9da9694d5070" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41215 } } peers { permanent_uuid: "5d9dad7486054aa99fa052291e222d90" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36481 } }
I20260924 10:10:42.404251 19547 tablet_bootstrap.cc:654] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:42.405633 19547 log.cc:824] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:42.414744 19547 tablet_bootstrap.cc:492] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3: No bootstrap required, opened a new log
I20260924 10:10:42.415102 19547 ts_tablet_manager.cc:1397] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20260924 10:10:42.437639 19520 heartbeater.cc:503] Master 127.6.2.254:38343 was elected leader, sending a full tablet report...
I20260924 10:10:42.439590 19550 raft_consensus.cc:482] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:10:42.439975 19550 raft_consensus.cc:504] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ee23ff3e48487fb01463de9f3170a3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40209 } } peers { permanent_uuid: "e48c6c0f641f4c1bbf7e9da9694d5070" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41215 } } peers { permanent_uuid: "5d9dad7486054aa99fa052291e222d90" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36481 } }
I20260924 10:10:42.444267 19547 raft_consensus.cc:348] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ee23ff3e48487fb01463de9f3170a3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40209 } } peers { permanent_uuid: "e48c6c0f641f4c1bbf7e9da9694d5070" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41215 } } peers { permanent_uuid: "5d9dad7486054aa99fa052291e222d90" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36481 } }
I20260924 10:10:42.444772 19547 raft_consensus.cc:374] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:42.444890 19547 raft_consensus.cc:729] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ee23ff3e48487fb01463de9f3170a3, State: Initialized, Role: FOLLOWER
I20260924 10:10:42.445933 19547 consensus_queue.cc:260] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ee23ff3e48487fb01463de9f3170a3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40209 } } peers { permanent_uuid: "e48c6c0f641f4c1bbf7e9da9694d5070" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41215 } } peers { permanent_uuid: "5d9dad7486054aa99fa052291e222d90" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36481 } }
I20260924 10:10:42.453806 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.443725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:40209 (local address 127.6.2.195:48905)
0924 10:10:42.443961 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:42.443983 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:42.444117 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:42.444827 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:42.444833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:42.444864 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:42.445091 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:42.445103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.446682 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.446687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:42.448743 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:42.448760 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.452696 (+ 3936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.452703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:42.452892 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.452927 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:10:42.453120 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 10:10:42.453277 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260924 10:10:42.453800 19547 ts_tablet_manager.cc:1428] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3: Time spent starting tablet: real 0.039s user 0.015s sys 0.000s
I20260924 10:10:42.454845 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.443374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:48905 (local address 127.6.2.193:40209)
0924 10:10:42.443574 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:42.443582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:42.444236 (+ 654us) server_negotiation.cc:407] Connection header received
0924 10:10:42.444323 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:42.444328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:42.444435 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:42.444522 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:42.445335 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.446445 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.449018 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.450647 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.450816 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.450968 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:42.454443 (+ 3475us) server_negotiation.cc:299] Negotiation successful
0924 10:10:42.454596 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:10:42.456355 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d3a5249b4d45e5b1b1823ce653f724" candidate_uuid: "5d9dad7486054aa99fa052291e222d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ee23ff3e48487fb01463de9f3170a3" is_pre_election: true
I20260924 10:10:42.456985 19196 raft_consensus.cc:2454] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9dad7486054aa99fa052291e222d90 in term 0.
I20260924 10:10:42.457789 19409 leader_election.cc:302] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9dad7486054aa99fa052291e222d90, 68ee23ff3e48487fb01463de9f3170a3; no voters:
I20260924 10:10:42.458600 19544 raft_consensus.cc:348] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ee23ff3e48487fb01463de9f3170a3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40209 } } peers { permanent_uuid: "e48c6c0f641f4c1bbf7e9da9694d5070" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41215 } } peers { permanent_uuid: "5d9dad7486054aa99fa052291e222d90" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36481 } }
I20260924 10:10:42.460182 19242 heartbeater.cc:503] Master 127.6.2.254:38343 was elected leader, sending a full tablet report...
I20260924 10:10:42.465651 19550 leader_election.cc:288] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68ee23ff3e48487fb01463de9f3170a3 (127.6.2.193:40209), e48c6c0f641f4c1bbf7e9da9694d5070 (127.6.2.194:41215)
I20260924 10:10:42.465901 19550 raft_consensus.cc:2781] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:10:42.466051 19550 raft_consensus.cc:482] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:10:42.466140 19550 raft_consensus.cc:3037] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:42.466786 19544 raft_consensus.cc:374] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:42.466928 19544 raft_consensus.cc:729] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48c6c0f641f4c1bbf7e9da9694d5070, State: Initialized, Role: FOLLOWER
I20260924 10:10:42.463284 19386 heartbeater.cc:503] Master 127.6.2.254:38343 was elected leader, sending a full tablet report...
I20260924 10:10:42.467680 19544 consensus_queue.cc:260] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ee23ff3e48487fb01463de9f3170a3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40209 } } peers { permanent_uuid: "e48c6c0f641f4c1bbf7e9da9694d5070" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41215 } } peers { permanent_uuid: "5d9dad7486054aa99fa052291e222d90" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36481 } }
W20260924 10:10:42.470824 19243 tablet.cc:2369] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:10:42.472334 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.459899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:41215 (local address 127.6.2.195:50257)
0924 10:10:42.460692 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:42.460714 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:42.460824 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:42.461355 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:42.461359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:42.461386 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:42.461621 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:42.461637 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.463007 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.463013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:42.465423 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:42.465440 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.471334 (+ 5894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.471340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:42.471618 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.471667 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 10:10:42.471837 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 10:10:42.472005 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673}
I20260924 10:10:42.473265 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.460455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:50257 (local address 127.6.2.194:41215)
0924 10:10:42.460733 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:42.460743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:42.460773 (+ 30us) server_negotiation.cc:407] Connection header received
0924 10:10:42.461005 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:42.461010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:42.461095 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:42.461184 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:42.461822 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.462839 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.465620 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.466946 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.467925 (+ 979us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.471153 (+ 3228us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:42.472656 (+ 1503us) server_negotiation.cc:299] Negotiation successful
0924 10:10:42.472926 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 10:10:42.472537 19550 raft_consensus.cc:504] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ee23ff3e48487fb01463de9f3170a3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40209 } } peers { permanent_uuid: "e48c6c0f641f4c1bbf7e9da9694d5070" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41215 } } peers { permanent_uuid: "5d9dad7486054aa99fa052291e222d90" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36481 } }
I20260924 10:10:42.474886 19341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d3a5249b4d45e5b1b1823ce653f724" candidate_uuid: "5d9dad7486054aa99fa052291e222d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48c6c0f641f4c1bbf7e9da9694d5070" is_pre_election: true
I20260924 10:10:42.473194 19543 ts_tablet_manager.cc:1428] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90: Time spent starting tablet: real 0.090s user 0.023s sys 0.058s
I20260924 10:10:42.475512 19341 raft_consensus.cc:2454] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9dad7486054aa99fa052291e222d90 in term 0.
I20260924 10:10:42.475953 19550 leader_election.cc:288] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [CANDIDATE]: Term 1 election: Requested vote from peers 68ee23ff3e48487fb01463de9f3170a3 (127.6.2.193:40209), e48c6c0f641f4c1bbf7e9da9694d5070 (127.6.2.194:41215)
I20260924 10:10:42.476307 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d3a5249b4d45e5b1b1823ce653f724" candidate_uuid: "5d9dad7486054aa99fa052291e222d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ee23ff3e48487fb01463de9f3170a3"
I20260924 10:10:42.476545 19196 raft_consensus.cc:3037] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:42.479498 19340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d3a5249b4d45e5b1b1823ce653f724" candidate_uuid: "5d9dad7486054aa99fa052291e222d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48c6c0f641f4c1bbf7e9da9694d5070"
I20260924 10:10:42.479761 19340 raft_consensus.cc:3037] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:42.481528 19196 raft_consensus.cc:2454] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9dad7486054aa99fa052291e222d90 in term 1.
I20260924 10:10:42.481992 19409 leader_election.cc:302] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9dad7486054aa99fa052291e222d90, 68ee23ff3e48487fb01463de9f3170a3; no voters:
I20260924 10:10:42.482389 19550 raft_consensus.cc:2781] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:10:42.482529 19550 raft_consensus.cc:686] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [term 1 LEADER]: Becoming Leader. State: Replica: 5d9dad7486054aa99fa052291e222d90, State: Running, Role: LEADER
I20260924 10:10:42.483089 19550 consensus_queue.cc:237] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ee23ff3e48487fb01463de9f3170a3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40209 } } peers { permanent_uuid: "e48c6c0f641f4c1bbf7e9da9694d5070" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41215 } } peers { permanent_uuid: "5d9dad7486054aa99fa052291e222d90" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36481 } }
I20260924 10:10:42.484539 19340 raft_consensus.cc:2454] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9dad7486054aa99fa052291e222d90 in term 1.
I20260924 10:10:42.486156 19544 ts_tablet_manager.cc:1428] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070: Time spent starting tablet: real 0.087s user 0.014s sys 0.001s
I20260924 10:10:42.494313 19045 catalog_manager.cc:5495] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d9dad7486054aa99fa052291e222d90 (127.6.2.195). New cstate: current_term: 1 leader_uuid: "5d9dad7486054aa99fa052291e222d90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ee23ff3e48487fb01463de9f3170a3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e48c6c0f641f4c1bbf7e9da9694d5070" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d9dad7486054aa99fa052291e222d90" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 36481 } health_report { overall_health: HEALTHY } } }
I20260924 10:10:42.536747 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.528722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.6.2.195:36481)
0924 10:10:42.529295 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:42.529299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:42.529314 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:10:42.529399 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:42.529403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:42.529485 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:42.529644 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:10:42.530022 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.530915 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.532985 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.533627 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.533747 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.534792 (+ 1045us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:10:42.535579 (+ 787us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:10:42.535723 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:42.536395 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 10:10:42.536474 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456}
I20260924 10:10:42.558411 19196 raft_consensus.cc:1261] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3 [term 1 FOLLOWER]: Refusing update from remote peer 5d9dad7486054aa99fa052291e222d90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:10:42.558491 19340 raft_consensus.cc:1261] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070 [term 1 FOLLOWER]: Refusing update from remote peer 5d9dad7486054aa99fa052291e222d90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:10:42.559566 19550 consensus_queue.cc:1035] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ee23ff3e48487fb01463de9f3170a3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:10:42.559926 19548 consensus_queue.cc:1035] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e48c6c0f641f4c1bbf7e9da9694d5070" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:10:42.575686 19564 mvcc.cc:205] Tried to move back new op lower bound from 7332842055906701312 to 7332842055623143424. Current Snapshot: MvccSnapshot[applied={T|T < 7332842055906701312}]
I20260924 10:10:42.579459 19555 mvcc.cc:205] Tried to move back new op lower bound from 7332842055906701312 to 7332842055623143424. Current Snapshot: MvccSnapshot[applied={T|T < 7332842055906701312 or (T in {7332842055906701312})}]
I20260924 10:10:42.589246 19045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:10:42.589545 19045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:10:42.593406 19045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7cecfc40a96472c82520c04835ba294: Sending DeleteTablet for 3 replicas of tablet 70d3a5249b4d45e5b1b1823ce653f724
I20260924 10:10:42.594620 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet 70d3a5249b4d45e5b1b1823ce653f724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:10:42 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:41314
I20260924 10:10:42.594694 19321 tablet_service.cc:1514] Processing DeleteTablet for tablet 70d3a5249b4d45e5b1b1823ce653f724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:10:42 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:58848
I20260924 10:10:42.595490 19455 tablet_service.cc:1514] Processing DeleteTablet for tablet 70d3a5249b4d45e5b1b1823ce653f724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:10:42 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:54086
I20260924 10:10:42.604844 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 19082
I20260924 10:10:42.607271 19572 tablet_replica.cc:330] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070: stopping tablet replica
I20260924 10:10:42.607846 19573 tablet_replica.cc:330] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90: stopping tablet replica
I20260924 10:10:42.607971 19572 raft_consensus.cc:2229] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:10:42.608223 19573 raft_consensus.cc:2229] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:10:42.608547 19572 raft_consensus.cc:2258] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:10:42.608934 19573 raft_consensus.cc:2258] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:10:42.610863 19572 ts_tablet_manager.cc:1905] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:10:42.612336 19573 ts_tablet_manager.cc:1905] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:10:42.617954 19572 ts_tablet_manager.cc:1918] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:10:42.618117 19572 log.cc:1196] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/70d3a5249b4d45e5b1b1823ce653f724
I20260924 10:10:42.618827 19571 tablet_replica.cc:330] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3: stopping tablet replica
I20260924 10:10:42.619299 19571 raft_consensus.cc:2229] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:10:42.619835 19571 raft_consensus.cc:2258] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:10:42.622164 19571 ts_tablet_manager.cc:1905] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:10:42.626112 19573 ts_tablet_manager.cc:1918] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:10:42.626698 19573 log.cc:1196] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/70d3a5249b4d45e5b1b1823ce653f724
I20260924 10:10:42.627496 19573 ts_tablet_manager.cc:1939] T 70d3a5249b4d45e5b1b1823ce653f724 P 5d9dad7486054aa99fa052291e222d90: Deleting consensus metadata
I20260924 10:10:42.629747 19031 catalog_manager.cc:4841] TS 5d9dad7486054aa99fa052291e222d90 (127.6.2.195:36481): tablet 70d3a5249b4d45e5b1b1823ce653f724 (table test-table [id=1b71eff145af444c9f48903459d6ed5c]) successfully deleted
I20260924 10:10:42.629814 19571 ts_tablet_manager.cc:1918] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:10:42.629971 19571 log.cc:1196] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/70d3a5249b4d45e5b1b1823ce653f724
I20260924 10:10:42.626554 19572 ts_tablet_manager.cc:1939] T 70d3a5249b4d45e5b1b1823ce653f724 P e48c6c0f641f4c1bbf7e9da9694d5070: Deleting consensus metadata
I20260924 10:10:42.630560 19571 ts_tablet_manager.cc:1939] T 70d3a5249b4d45e5b1b1823ce653f724 P 68ee23ff3e48487fb01463de9f3170a3: Deleting consensus metadata
I20260924 10:10:42.632443 19029 catalog_manager.cc:4841] TS e48c6c0f641f4c1bbf7e9da9694d5070 (127.6.2.194:41215): tablet 70d3a5249b4d45e5b1b1823ce653f724 (table test-table [id=1b71eff145af444c9f48903459d6ed5c]) successfully deleted
I20260924 10:10:42.632575 19030 catalog_manager.cc:4841] TS 68ee23ff3e48487fb01463de9f3170a3 (127.6.2.193:40209): tablet 70d3a5249b4d45e5b1b1823ce653f724 (table test-table [id=1b71eff145af444c9f48903459d6ed5c]) successfully deleted
I20260924 10:10:42.633545 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.606564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.6.2.193:40209)
0924 10:10:42.606770 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:42.606779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:42.606806 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:10:42.606894 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:42.606901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:42.607028 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:42.607166 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:42.611038 (+ 3872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.612804 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.617936 (+ 5132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.618873 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.618970 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.624103 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:42.624119 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:42.624138 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:42.624195 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:42.628968 (+ 4773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:42.630640 (+ 1672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:42.630655 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:42.630662 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:42.630850 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:42.631402 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:42.631408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:42.631411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:42.631904 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:42.632224 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:42.632956 (+ 732us) server_negotiation.cc:299] Negotiation successful
0924 10:10:42.633183 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:10:42.634199 19237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:42.731869 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 19082
I20260924 10:10:42.749675 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 19246
I20260924 10:10:42.765354 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.750487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.6.2.194:41215)
0924 10:10:42.750656 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:42.750661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:42.750681 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:10:42.750947 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:42.750951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:42.751031 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:42.751195 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:42.752921 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.753614 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.755497 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.755902 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.755997 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.759230 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:42.759246 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:42.759264 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:42.759299 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:42.762370 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:42.763287 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:42.763299 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:42.763305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:42.763450 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:42.763834 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:42.763837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:42.763839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:42.764232 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:42.764455 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:42.764818 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 10:10:42.765042 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:10:42.766077 19381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:42.859647 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 19246
I20260924 10:10:42.879226 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 19388
I20260924 10:10:42.896667 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:42.880050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.6.2.195:36481)
0924 10:10:42.880246 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:42.880251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:42.880328 (+ 77us) server_negotiation.cc:407] Connection header received
0924 10:10:42.880523 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:42.880527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:42.880600 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:42.880715 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:42.882595 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.883718 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.885654 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:42.886149 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:42.886247 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:42.889856 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:42.889872 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:42.889889 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:42.889934 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:42.893399 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:42.894306 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:42.894314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:42.894316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:42.894470 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:42.894905 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:42.894909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:42.894911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:42.895348 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:42.895809 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:42.896103 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 10:10:42.896293 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:10:42.897409 19515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:43.021184 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 19388
I20260924 10:10:43.041679 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 19012
I20260924 10:10:43.042862 19075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:43.156469 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 19012
I20260924 10:10:43.179499 6155 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 10:10:43.516018 6155 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 10:10:43.516228 6155 mini_ranger.cc:66] Stopping Ranger...
I20260924 10:10:43.862738 6155 mini_ranger.cc:68] Stopped Ranger
2026-09-24 10:10:43.863 UTC [18064] LOG: received smart shutdown request
2026-09-24 10:10:43.866 UTC [18064] LOG: background worker "logical replication launcher" (PID 18073) exited with exit code 1
2026-09-24 10:10:43.866 UTC [18068] LOG: shutting down
2026-09-24 10:10:44.088 UTC [18064] LOG: database system is shut down
2026-09-24T10:10:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54292 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:10:44 dist-test-slave-s3n3 krb5kdc[19586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:10:44 dist-test-slave-s3n3 krb5kdc[19586](info): set up 2 sockets
Sep 24 10:10:44 dist-test-slave-s3n3 krb5kdc[19586](info): commencing operation
krb5kdc: starting...
W20260924 10:10:45.343593 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:10:45 dist-test-slave-s3n3 krb5kdc[19586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:10:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:10:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:10:45.583796 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34039
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:34039,127.6.2.253:34385,127.6.2.252:33143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:10:45.920701 19599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:45.921128 19599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:45.921317 19599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:45.929322 19599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:10:45.929499 19599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:45.929608 19599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:45.929714 19599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:10:45.929802 19599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:10:45.940658 19599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:34039,127.6.2.253:34385,127.6.2.252:33143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:45.942131 19599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:45.943478 19599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:45.955574 19604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:45.955729 19607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:45.955546 19605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:45.956336 19599 server_base.cc:1034] running on GCE node
I20260924 10:10:45.957105 19599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:45.958601 19599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:45.959832 19599 hybrid_clock.cc:648] HybridClock initialized: now 1790244645959783 us; error 47 us; skew 500 ppm
I20260924 10:10:45.960227 19599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:45 dist-test-slave-s3n3 krb5kdc[19586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244645, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:45.963912 19599 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:10:45.966032 19599 webserver.cc:460] Webserver started at http://127.6.2.254:42571/ using document root <none> and password file <none>
I20260924 10:10:45.966955 19599 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:45.967257 19599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:45.967664 19599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:10:45.971122 19599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "147cc9732da04f84a7e599389559cf49"
format_stamp: "Formatted at 2026-09-24 10:10:45 on dist-test-slave-s3n3"
server_key: "f68b6d90940fa1507baac8dd2b78f14f"
server_key_iv: "0085a0a7b59bb04cc52ec222980b1add"
server_key_version: "encryptionkey@0"
I20260924 10:10:45.971992 19599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "147cc9732da04f84a7e599389559cf49"
format_stamp: "Formatted at 2026-09-24 10:10:45 on dist-test-slave-s3n3"
server_key: "f68b6d90940fa1507baac8dd2b78f14f"
server_key_iv: "0085a0a7b59bb04cc52ec222980b1add"
server_key_version: "encryptionkey@0"
I20260924 10:10:45.978420 19599 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 10:10:45.982461 19614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:45.984187 19599 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 10:10:45.984388 19599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "147cc9732da04f84a7e599389559cf49"
format_stamp: "Formatted at 2026-09-24 10:10:45 on dist-test-slave-s3n3"
server_key: "f68b6d90940fa1507baac8dd2b78f14f"
server_key_iv: "0085a0a7b59bb04cc52ec222980b1add"
server_key_version: "encryptionkey@0"
I20260924 10:10:45.984599 19599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:45.997939 19599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:46.002463 19599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:46.002785 19599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:46.022761 19599 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:34039
I20260924 10:10:46.022786 19665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:34039 every 8 connection(s)
I20260924 10:10:46.024262 19599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:10:46.026857 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 19599
I20260924 10:10:46.027139 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:10:46.027804 6155 external_mini_cluster.cc:1428] Setting key dca147babe258b7a5180e2f70152db65
I20260924 10:10:46.028287 19666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:46.034955 19666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.2.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.253@KRBTEST.COM" created.
Sep 24 10:10:46 dist-test-slave-s3n3 krb5kdc[19586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244645, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:46.059494 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.038539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34039 (local address 127.0.0.1:41332)
0924 10:10:46.039409 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.039456 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:46.041629 (+ 2173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:46.042569 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:46.042574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:46.043334 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:46.043668 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:46.043685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.046242 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.046246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:46.047550 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:46.047568 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.048616 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.048622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:46.048755 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:46.049444 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:46.049471 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:46.051829 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:46.055209 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:46.055222 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:46.055235 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:46.055624 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:46.056052 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:46.056059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:46.056062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:46.056213 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:46.056979 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:46.056992 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:46.057210 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 10:10:46.057455 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 10:10:46.058099 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":170,"thread_start_us":388,"threads_started":1}
I20260924 10:10:46.059517 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.039873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.6.2.254:34039)
0924 10:10:46.040685 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:46.040696 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:46.040762 (+ 66us) server_negotiation.cc:407] Connection header received
0924 10:10:46.042015 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:46.042026 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:46.042321 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:46.042546 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:46.043881 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.046033 (+ 2152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.047695 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.048346 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.048483 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:46.052064 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:46.052072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:46.052084 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:46.052116 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:46.054994 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:46.055774 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:46.055781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:46.055783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:46.055940 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:46.056334 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:46.056338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:46.056341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:46.056845 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:46.057063 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:46.057588 (+ 525us) server_negotiation.cc:299] Negotiation successful
0924 10:10:46.057988 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":573,"thread_start_us":368,"threads_started":1}
I20260924 10:10:46.062393 19666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:10:46.063555 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.063158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:34385 (local address 127.0.0.1:53326)
0924 10:10:46.063333 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.063443 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 10:10:46.063830 19617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
I20260924 10:10:46.065649 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.065174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:34385 (local address 127.0.0.1:53334)
0924 10:10:46.065343 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.065442 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 10:10:46.065837 19666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:34385: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.253.keytab.
Entry for principal kudu/127.6.2.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.253.keytab.
I20260924 10:10:46.080291 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.253:34385
--webserver_interface=127.6.2.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.253.keytab
--principal=kudu/127.6.2.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:34039,127.6.2.253:34385,127.6.2.252:33143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:10:46.115334 19666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } attempt: 1
I20260924 10:10:46.117259 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.116618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:34385 (local address 127.0.0.1:53350)
0924 10:10:46.116903 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.117062 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 10:10:46.119328 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.118362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:34385 (local address 127.0.0.1:53360)
0924 10:10:46.118658 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.118828 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260924 10:10:46.119647 19666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:34385: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
I20260924 10:10:46.187961 19666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } attempt: 2
I20260924 10:10:46.189189 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.188692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:34385 (local address 127.0.0.1:53364)
0924 10:10:46.188955 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.189054 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 10:10:46.190369 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.189997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:34385 (local address 127.0.0.1:53370)
0924 10:10:46.190165 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.190236 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 10:10:46.190614 19666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:34385: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
I20260924 10:10:46.281900 19666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } attempt: 3
I20260924 10:10:46.283305 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.282779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:34385 (local address 127.0.0.1:53378)
0924 10:10:46.283004 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.283163 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 10:10:46.284430 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.284066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:34385 (local address 127.0.0.1:53392)
0924 10:10:46.284225 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.284294 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 10:10:46.284683 19666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:34385: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
W20260924 10:10:46.423822 19673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:46.424113 19673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:46.424209 19673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 10:10:46.427944 19666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } attempt: 4
I20260924 10:10:46.429119 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.428598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:34385 (local address 127.0.0.1:60704)
0924 10:10:46.428904 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.428994 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260924 10:10:46.430140 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.429810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:34385 (local address 127.0.0.1:60716)
0924 10:10:46.429961 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.430024 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 10:10:46.430370 19666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:34385: Network error: Client connection negotiation failed: client connection to 127.6.2.253:34385: connect: Connection refused (error 111)
W20260924 10:10:46.431834 19673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:10:46.431931 19673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:46.431975 19673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:46.432047 19673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:10:46.432130 19673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:10:46.442142 19673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:34039,127.6.2.253:34385,127.6.2.252:33143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.253.keytab
--principal=kudu/127.6.2.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.253:34385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.2.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:46.443513 19673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:46.444802 19673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:46.456615 19681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:46.456763 19678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:46.456733 19679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:46.457567 19673 server_base.cc:1034] running on GCE node
I20260924 10:10:46.458278 19673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:46.459744 19673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:46.460954 19673 hybrid_clock.cc:648] HybridClock initialized: now 1790244646460920 us; error 32 us; skew 500 ppm
I20260924 10:10:46.461318 19673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:46 dist-test-slave-s3n3 krb5kdc[19586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244646, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:46.470287 19673 init.cc:377] Logged in from keytab as kudu/127.6.2.253@KRBTEST.COM (short username kudu)
I20260924 10:10:46.472018 19673 webserver.cc:460] Webserver started at http://127.6.2.253:46613/ using document root <none> and password file <none>
I20260924 10:10:46.472754 19673 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:46.472910 19673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:46.473237 19673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:10:46.476471 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/data/instance:
uuid: "ebb0a90d3a0443048315ce762c753c31"
format_stamp: "Formatted at 2026-09-24 10:10:46 on dist-test-slave-s3n3"
server_key: "69e05ad9a1630eb623d1e78dc014e232"
server_key_iv: "25bfc369d0a4a5fb8a76b5e3418b2c8e"
server_key_version: "encryptionkey@0"
I20260924 10:10:46.477344 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/wal/instance:
uuid: "ebb0a90d3a0443048315ce762c753c31"
format_stamp: "Formatted at 2026-09-24 10:10:46 on dist-test-slave-s3n3"
server_key: "69e05ad9a1630eb623d1e78dc014e232"
server_key_iv: "25bfc369d0a4a5fb8a76b5e3418b2c8e"
server_key_version: "encryptionkey@0"
I20260924 10:10:46.482555 19673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 10:10:46.486543 19688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:46.488467 19673 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:10:46.488674 19673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/wal
uuid: "ebb0a90d3a0443048315ce762c753c31"
format_stamp: "Formatted at 2026-09-24 10:10:46 on dist-test-slave-s3n3"
server_key: "69e05ad9a1630eb623d1e78dc014e232"
server_key_iv: "25bfc369d0a4a5fb8a76b5e3418b2c8e"
server_key_version: "encryptionkey@0"
I20260924 10:10:46.488940 19673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:46.515933 19673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:46.521005 19673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:46.521344 19673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:46.540531 19673 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.253:34385
I20260924 10:10:46.540565 19739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.253:34385 every 8 connection(s)
I20260924 10:10:46.541917 19673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/data/info.pb
I20260924 10:10:46.543788 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 19673
I20260924 10:10:46.543990 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-1/wal/instance
I20260924 10:10:46.544463 6155 external_mini_cluster.cc:1428] Setting key 43ca70f38b49249c09fbcda7ea3ec818
I20260924 10:10:46.546032 19740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:46.553400 19740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.2.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.252@KRBTEST.COM" created.
Sep 24 10:10:46 dist-test-slave-s3n3 krb5kdc[19586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244646, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.253@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:46.575362 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.556720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.6.2.254:34039)
0924 10:10:46.556991 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:46.556999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:46.558223 (+ 1224us) server_negotiation.cc:407] Connection header received
0924 10:10:46.560172 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:46.560181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:46.560282 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:46.560432 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:46.562019 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.563170 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.564797 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.565358 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.565459 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:46.569183 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:46.569188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:46.569192 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:46.569225 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:46.572199 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:46.572975 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:46.572982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:46.572991 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:46.573108 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:46.573531 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:46.573537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:46.573540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:46.573851 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:46.574018 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:46.574740 (+ 722us) server_negotiation.cc:299] Negotiation successful
0924 10:10:46.574968 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 10:10:46.576365 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.557175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34039 (local address 127.0.0.1:37002)
0924 10:10:46.558039 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.558090 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:46.559897 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:46.560661 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:46.560669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:46.561269 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:46.561832 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:46.561849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.563347 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.563351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:46.564644 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:46.564657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.565564 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.565569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:46.565760 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:46.566479 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:46.566506 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:46.568937 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:46.572362 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:46.572372 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:46.572390 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:46.572836 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:46.573224 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:46.573238 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:46.573242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:46.573414 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:46.573990 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:46.573998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:46.574370 (+ 372us) client_negotiation.cc:769] Sending connection context
0924 10:10:46.574645 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 10:10:46.575219 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":390,"threads_started":1}
I20260924 10:10:46.578251 19740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:10:46 dist-test-slave-s3n3 krb5kdc[19586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244646, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.253@KRBTEST.COM for kudu/127.6.2.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.252.keytab.
Entry for principal kudu/127.6.2.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.252.keytab.
I20260924 10:10:46.596630 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.579277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:34385 (local address 127.0.0.1:60730)
0924 10:10:46.579522 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.579546 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:46.579659 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:46.580744 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:46.580753 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:46.581267 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:46.581480 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:46.581495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.582906 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.582914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:46.583822 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:46.583834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.584659 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.584664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:46.584971 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:46.585634 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:46.585653 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:46.589990 (+ 4337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:46.593703 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:46.593707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:46.593710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:46.594112 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:46.594602 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:46.594606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:46.594608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:46.594808 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:46.595598 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:46.595606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:46.595764 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 10:10:46.595905 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 10:10:46.596209 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":5}
I20260924 10:10:46.596689 19745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.579547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.6.2.253:34385)
0924 10:10:46.580242 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:46.580248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:46.580284 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:10:46.580329 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:46.580334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:46.580515 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:46.580633 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:46.581650 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.582742 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.583963 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.584512 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.584650 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:46.590219 (+ 5569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:46.590232 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:46.590246 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:46.590275 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:46.593567 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:46.594309 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:46.594316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:46.594319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:46.594450 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:46.594939 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:46.594944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:46.594947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:46.595425 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:46.595694 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:46.596060 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 10:10:46.596318 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":549,"thread_start_us":412,"threads_started":1}
I20260924 10:10:46.596755 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.252:33143
--webserver_interface=127.6.2.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.252.keytab
--principal=kudu/127.6.2.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:34039,127.6.2.253:34385,127.6.2.252:33143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:10:46.599323 19740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:10:46.600421 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.599973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35752)
0924 10:10:46.600137 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.600251 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 10:10:46.600657 19690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
I20260924 10:10:46.602173 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.601849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35754)
0924 10:10:46.601998 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.602072 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:10:46.602408 19740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:33143: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
I20260924 10:10:46.651670 19740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } attempt: 1
I20260924 10:10:46.652849 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.652412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35758)
0924 10:10:46.652611 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.652707 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 10:10:46.653767 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.653445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35772)
0924 10:10:46.653614 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.653675 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 10:10:46.653970 19740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:33143: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
I20260924 10:10:46.722230 19740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } attempt: 2
I20260924 10:10:46.723443 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.722936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35782)
0924 10:10:46.723176 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.723275 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 10:10:46.724385 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.724075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35788)
0924 10:10:46.724213 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.724282 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:10:46.724567 19740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:33143: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
I20260924 10:10:46.729662 19666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } attempt: 5
Sep 24 10:10:46 dist-test-slave-s3n3 krb5kdc[19586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244645, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for kudu/127.6.2.253@KRBTEST.COM
I20260924 10:10:46.744273 19745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.730400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.6.2.253:34385)
0924 10:10:46.730556 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:46.730560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:46.730636 (+ 76us) server_negotiation.cc:407] Connection header received
0924 10:10:46.730870 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:46.730874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:46.730935 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:46.731019 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:46.732044 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.732815 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.734600 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.735120 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.735246 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:46.738915 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:46.738920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:46.738923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:46.738954 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:46.741648 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:46.742307 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:46.742314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:46.742320 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:46.742445 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:46.742823 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:46.742831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:46.742834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:46.743173 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:46.743341 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:46.743720 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 10:10:46.743929 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:10:46.744295 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.730325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:34385 (local address 127.0.0.1:60740)
0924 10:10:46.730532 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.730553 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:46.730699 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:46.731111 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:46.731116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:46.731640 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:46.731870 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:46.731884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.732963 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.732969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:46.734394 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:46.734431 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:46.735299 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:46.735317 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:46.735532 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:46.736169 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:46.736192 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:46.738704 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:46.741797 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:46.741803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:46.741807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:46.742179 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:46.742559 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:46.742563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:46.742567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:46.742707 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:46.743308 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:46.743313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:46.743468 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 10:10:46.743613 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 10:10:46.743917 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 10:10:46.745831 19666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:10:46.746857 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.746452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35796)
0924 10:10:46.746612 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.746690 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 10:10:46.747792 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.747486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35804)
0924 10:10:46.747633 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.747703 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:10:46.748014 19666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:33143: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
I20260924 10:10:46.799288 19666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } attempt: 1
I20260924 10:10:46.800529 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.799955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35808)
0924 10:10:46.800193 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.800326 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 10:10:46.801627 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.801255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35818)
0924 10:10:46.801412 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.801494 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 10:10:46.801923 19666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:33143: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
I20260924 10:10:46.815800 19740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } attempt: 3
I20260924 10:10:46.817055 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.816494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35830)
0924 10:10:46.816747 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.816861 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 10:10:46.818307 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.817725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35832)
0924 10:10:46.818002 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.818116 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260924 10:10:46.818567 19740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:33143: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
I20260924 10:10:46.870191 19666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } attempt: 2
I20260924 10:10:46.871495 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.870925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35842)
0924 10:10:46.871191 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.871298 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 10:10:46.872642 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.872246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35852)
0924 10:10:46.872404 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.872488 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 10:10:46.872880 19666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:33143: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
W20260924 10:10:46.936713 19747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:46.937028 19747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:46.937177 19747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:46.944655 19747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:10:46.944778 19747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:46.944865 19747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:46.944972 19747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:10:46.945042 19747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:10:46.955396 19747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:34039,127.6.2.253:34385,127.6.2.252:33143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.252.keytab
--principal=kudu/127.6.2.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.252:33143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.2.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:46.956831 19747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:46.958716 19747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:10:46.961807 19740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } attempt: 4
I20260924 10:10:46.963009 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.962524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35868)
0924 10:10:46.962766 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.962851 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 10:10:46.964154 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.963734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35876)
0924 10:10:46.963888 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.963989 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 10:10:46.964397 19740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:33143: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
W20260924 10:10:46.971606 19753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:46.971628 19755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:46.972432 19747 server_base.cc:1034] running on GCE node
W20260924 10:10:46.972570 19752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:46.973459 19747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:46.974968 19747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:46.976250 19747 hybrid_clock.cc:648] HybridClock initialized: now 1790244646976201 us; error 86 us; skew 500 ppm
I20260924 10:10:46.976643 19747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:10:46.979167 19666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } attempt: 3
Sep 24 10:10:46 dist-test-slave-s3n3 krb5kdc[19586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244646, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:46.980363 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.979749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35880)
0924 10:10:46.980030 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.980144 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 10:10:46.980556 19747 init.cc:377] Logged in from keytab as kudu/127.6.2.252@KRBTEST.COM (short username kudu)
I20260924 10:10:46.981637 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:46.981226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35894)
0924 10:10:46.981446 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:46.981514 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260924 10:10:46.981986 19666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:33143: Network error: Client connection negotiation failed: client connection to 127.6.2.252:33143: connect: Connection refused (error 111)
I20260924 10:10:46.982681 19747 webserver.cc:460] Webserver started at http://127.6.2.252:36393/ using document root <none> and password file <none>
I20260924 10:10:46.983559 19747 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:46.983676 19747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:46.984007 19747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:10:46.987478 19747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/data/instance:
uuid: "24d71128e20e41c7a5357575c010859e"
format_stamp: "Formatted at 2026-09-24 10:10:46 on dist-test-slave-s3n3"
server_key: "587ca86246ac2aadd7ed018ecbb1a8f1"
server_key_iv: "8e280a4914cecd806e5e29e5171081cf"
server_key_version: "encryptionkey@0"
I20260924 10:10:46.988394 19747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/wal/instance:
uuid: "24d71128e20e41c7a5357575c010859e"
format_stamp: "Formatted at 2026-09-24 10:10:46 on dist-test-slave-s3n3"
server_key: "587ca86246ac2aadd7ed018ecbb1a8f1"
server_key_iv: "8e280a4914cecd806e5e29e5171081cf"
server_key_version: "encryptionkey@0"
I20260924 10:10:46.994292 19747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:10:46.997943 19762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:46.999603 19747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:10:46.999755 19747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/wal
uuid: "24d71128e20e41c7a5357575c010859e"
format_stamp: "Formatted at 2026-09-24 10:10:46 on dist-test-slave-s3n3"
server_key: "587ca86246ac2aadd7ed018ecbb1a8f1"
server_key_iv: "8e280a4914cecd806e5e29e5171081cf"
server_key_version: "encryptionkey@0"
I20260924 10:10:46.999935 19747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:47.013149 19747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:47.019383 19747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:47.019642 19747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:47.042799 19747 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.252:33143
I20260924 10:10:47.042827 19813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.252:33143 every 8 connection(s)
I20260924 10:10:47.044332 19747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/data/info.pb
I20260924 10:10:47.048336 19814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:47.050593 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 19747
I20260924 10:10:47.050818 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790244437048964-6155-0/minicluster-data/master-2/wal/instance
I20260924 10:10:47.051412 6155 external_mini_cluster.cc:1428] Setting key 725682486c860087fdc72ba4e19b82db
I20260924 10:10:47.057587 19814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:10:47 dist-test-slave-s3n3 krb5kdc[19586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.252@KRBTEST.COM
Sep 24 10:10:47 dist-test-slave-s3n3 krb5kdc[19586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244646, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.252@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:47.074919 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.060322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37018 (local address 127.6.2.254:34039)
0924 10:10:47.060506 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:47.060532 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:47.061120 (+ 588us) server_negotiation.cc:407] Connection header received
0924 10:10:47.061352 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:47.061358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:47.061429 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:47.061542 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:47.062615 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.063363 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.064802 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.065236 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.065375 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.068973 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:47.068982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:47.068987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:47.069031 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:47.071831 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.072542 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.072550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.072555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.072722 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.073119 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.073124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.073127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.073406 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:47.073584 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:47.074315 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 10:10:47.074547 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:10:47.076108 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.060257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34039 (local address 127.0.0.1:37018)
0924 10:10:47.060976 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:47.061019 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:47.061220 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:47.061595 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:47.061603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:47.062195 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:47.062456 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:47.062465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.063535 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.063539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:47.064648 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:47.064661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.065467 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.065475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:47.065631 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.066268 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:47.066290 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:47.068737 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:47.071981 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:47.071994 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:47.072001 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:47.072420 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:47.072839 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:47.072845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:47.072849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:47.072995 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:47.073662 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:47.073670 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:47.073926 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 10:10:47.075141 (+ 1215us) client_negotiation.cc:240] Negotiation successful
0924 10:10:47.075446 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":1,"spinlock_wait_cycles":17280,"thread_start_us":395,"threads_started":1}
I20260924 10:10:47.076128 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.054278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.6.2.252:33143)
0924 10:10:47.055280 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:47.055293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:47.055358 (+ 65us) server_negotiation.cc:407] Connection header received
0924 10:10:47.057044 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:47.057064 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:47.057457 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:47.057880 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:47.059445 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.061449 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.062841 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.063552 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.063780 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.068385 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:47.068394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:47.068406 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:47.068443 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:47.071238 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.071934 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.071943 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.071959 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.072122 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.072514 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.072519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.072522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.073105 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:47.073397 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:47.074065 (+ 668us) server_negotiation.cc:299] Negotiation successful
0924 10:10:47.074647 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"spinlock_wait_cycles":20352,"thread_start_us":372,"threads_started":1}
I20260924 10:10:47.077932 19814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:10:47 dist-test-slave-s3n3 krb5kdc[19586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244646, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.252@KRBTEST.COM for kudu/127.6.2.253@KRBTEST.COM
Sep 24 10:10:47 dist-test-slave-s3n3 krb5kdc[19586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:47.094013 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.078807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:34385 (local address 127.0.0.1:60748)
0924 10:10:47.079040 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:47.079090 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:47.079244 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:47.079761 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:47.079769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:47.080243 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:47.080455 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:47.080469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.082066 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.082070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:47.083242 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:47.083257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.084009 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.084015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:47.084234 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.085222 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:47.085247 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:47.087852 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:47.091329 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:47.091335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:47.091338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:47.091786 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:47.092180 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:47.092186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:47.092190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:47.092341 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:47.093073 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:47.093078 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:47.093248 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 10:10:47.093398 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 10:10:47.093697 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260924 10:10:47.094159 19745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.078767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.6.2.253:34385)
0924 10:10:47.078951 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:47.078956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:47.079176 (+ 220us) server_negotiation.cc:407] Connection header received
0924 10:10:47.079379 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:47.079384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:47.079474 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:47.079574 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:47.080772 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.081901 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.083411 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.083859 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.083957 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.088042 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:47.088047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:47.088050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:47.088080 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:47.091193 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.091936 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.091940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.091942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.092066 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.092484 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.092487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.092489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.092811 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:47.093004 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:47.093517 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 10:10:47.093762 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:10:47.095049 19814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:10:47.095917 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.079555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.6.2.254:34039)
0924 10:10:47.079700 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:47.079704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:47.079719 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:10:47.079834 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:47.079838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:47.079928 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:47.080004 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:47.081436 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.082536 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.084159 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.084702 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.084830 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.089963 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:47.089971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:47.089978 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:47.090037 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:47.092762 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.093618 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.093624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.093627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.093765 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.094283 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.094288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.094291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.094652 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:47.094929 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:47.095321 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 10:10:47.095556 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 10:10:47 dist-test-slave-s3n3 krb5kdc[19586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244646, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.252@KRBTEST.COM for kudu/127.6.2.252@KRBTEST.COM
Sep 24 10:10:47 dist-test-slave-s3n3 krb5kdc[19586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.253@KRBTEST.COM
I20260924 10:10:47.110834 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.096040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35908 (local address 127.6.2.252:33143)
0924 10:10:47.096182 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:47.096187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:47.096463 (+ 276us) server_negotiation.cc:407] Connection header received
0924 10:10:47.096632 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:47.096637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:47.096701 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:47.096804 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:47.097739 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.098837 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.100309 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.100826 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.100954 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.104620 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:47.104627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:47.104634 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:47.104677 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:47.107647 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.108440 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.108445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.108447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.108583 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.109015 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.109019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.109021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.109528 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:47.109651 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:47.110233 (+ 582us) server_negotiation.cc:299] Negotiation successful
0924 10:10:47.110585 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":192,"server-negotiator.queue_time_us":50}
I20260924 10:10:47.110927 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.095838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35908)
0924 10:10:47.095999 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:47.096017 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:47.096523 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:47.096906 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:47.096914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:47.097360 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:47.097539 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:47.097550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.098978 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.098982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:47.100171 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:47.100184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.100965 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.100969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:47.101176 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.101764 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:47.101781 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:47.104406 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:47.107767 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:47.107773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:47.107778 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:47.108285 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:47.108707 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:47.108713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:47.108716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:47.108879 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:47.109759 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:47.109767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:47.109947 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 10:10:47.110111 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 10:10:47.110452 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":34}
I20260924 10:10:47.113933 19745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.097495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.6.2.253:34385)
0924 10:10:47.097751 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:47.097756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:47.097770 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:10:47.097944 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:47.097948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:47.098000 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:47.098071 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:47.099536 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.100570 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.101931 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.102579 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.102705 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.107514 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:47.107519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:47.107522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:47.107552 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:47.111246 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.111920 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.111927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.111930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.112063 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.112450 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.112455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.112458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.112728 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:47.113018 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:47.113270 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 10:10:47.113600 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 10:10:47.116479 19814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e: Bootstrap starting.
I20260924 10:10:47.120234 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:10:47.120666 19814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:47.121704 19814 log.cc:824] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:47.124800 19814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e: No bootstrap required, opened a new log
I20260924 10:10:47.140455 19814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } }
I20260924 10:10:47.140939 19814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:47.141062 19814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24d71128e20e41c7a5357575c010859e, State: Initialized, Role: FOLLOWER
I20260924 10:10:47.142210 19814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } }
I20260924 10:10:47.148025 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.126699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37030 (local address 127.6.2.254:34039)
0924 10:10:47.126894 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:47.126899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:47.126919 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:10:47.126984 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:47.126991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:47.127108 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:47.127222 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:47.130020 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.131505 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.134608 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.135559 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.135696 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.140775 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:47.140783 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:47.140801 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:47.140855 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:47.144502 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.145465 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.145473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.145479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.145644 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.146121 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.146127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.146130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.146482 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:47.146716 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:47.147333 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 10:10:47.147579 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:10:47.149427 19827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } } }
I20260924 10:10:47.149796 19827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:10:47.151515 19814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:10:47.155972 19745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.126409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.6.2.253:34385)
0924 10:10:47.126911 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:47.126918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:47.137858 (+ 10940us) server_negotiation.cc:407] Connection header received
0924 10:10:47.138053 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:47.138058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:47.138146 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:47.138243 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:47.140425 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.141478 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.143672 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.144612 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.144744 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.149267 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:47.149273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:47.149277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:47.149320 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:47.153067 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.153810 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.153818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.153823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.153999 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.154403 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.154408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.154411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.154704 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:47.154962 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:47.155392 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 10:10:47.155670 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377}
I20260924 10:10:47.159256 19666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } attempt: 4
I20260924 10:10:47.162418 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.127801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.6.2.252:33143)
0924 10:10:47.127988 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:47.127996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:47.139391 (+ 11395us) server_negotiation.cc:407] Connection header received
0924 10:10:47.139672 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:47.139680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:47.139771 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:47.139842 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:47.142284 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.143447 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.146127 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.146877 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.146969 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.152615 (+ 5646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:47.152628 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:47.152636 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:47.152684 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:47.157410 (+ 4726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.159297 (+ 1887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.159307 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.159314 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.159636 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.160147 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.160156 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.160159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.160598 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:47.161838 (+ 1240us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:47.161979 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 10:10:47.162145 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 10:10:47 dist-test-slave-s3n3 krb5kdc[19586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244645, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for kudu/127.6.2.252@KRBTEST.COM
W20260924 10:10:47.178565 19839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:10:47.178709 19839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:10:47.182070 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.159894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35934)
0924 10:10:47.160150 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:47.160175 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:47.160316 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:47.161640 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:47.161647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:47.162276 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:47.162570 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:47.162587 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.165152 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.165161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:47.167162 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:47.167184 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.169803 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.169812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:47.170094 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.170946 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:47.170974 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:47.174780 (+ 3806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:47.179525 (+ 4745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:47.179531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:47.179536 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:47.179911 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:47.180295 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:47.180301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:47.180305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:47.180441 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:47.181043 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:47.181047 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:47.181207 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 10:10:47.181356 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 10:10:47.181691 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 10:10:47.182132 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.160396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.6.2.252:33143)
0924 10:10:47.161217 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:47.161225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:47.161252 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:10:47.161321 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:47.161329 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:47.161420 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:47.162905 (+ 1485us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:47.163354 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.164951 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.167414 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.169605 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.169748 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.175022 (+ 5274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:47.175030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:47.175034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:47.175100 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:47.179366 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.180045 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.180048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.180050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.180170 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.180552 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.180560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.180566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.180893 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:47.181084 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:47.181452 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 10:10:47.181693 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":414,"threads_started":1}
I20260924 10:10:47.174688 19747 master_runner.cc:186] Error getting master registration for 127.6.2.254:34039: OK, instance_id { permanent_uuid: "147cc9732da04f84a7e599389559cf49" instance_seqno: 1790244646016972 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:10:47.188680 19666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49: Bootstrap starting.
I20260924 10:10:47.190009 19747 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:10:47.193437 19666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:47.194550 19666 log.cc:824] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:47.197578 19666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49: No bootstrap required, opened a new log
I20260924 10:10:47.206019 19666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } }
I20260924 10:10:47.206369 19666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:47.206506 19666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147cc9732da04f84a7e599389559cf49, State: Initialized, Role: FOLLOWER
I20260924 10:10:47.207124 19666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } }
I20260924 10:10:47.208115 19842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } } }
I20260924 10:10:47.208586 19842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:10:47.209113 19666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 10:10:47.220978 19853 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:10:47.221194 19853 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:10:47.220642 19599 master_runner.cc:186] Error getting master registration for 127.6.2.253:34385: OK, instance_id { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" instance_seqno: 1790244646534868 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:10:47.231439 19599 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 10:10:47.255838 19620 tablet.cc:2369] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:10:47.263779 19740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } attempt: 5
W20260924 10:10:47.272246 19768 tablet.cc:2369] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:10:47 dist-test-slave-s3n3 krb5kdc[19586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244646, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.253@KRBTEST.COM for kudu/127.6.2.252@KRBTEST.COM
I20260924 10:10:47.279278 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.264755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.6.2.252:33143)
0924 10:10:47.264959 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:47.264967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:47.265071 (+ 104us) server_negotiation.cc:407] Connection header received
0924 10:10:47.265269 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:47.265275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:47.265367 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:47.265476 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:47.266404 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.267242 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.268948 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.269618 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.269731 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.273658 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:47.273664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:47.273669 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:47.273713 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:47.276558 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.277259 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.277266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.277270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.277408 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:47.277810 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:47.277816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:47.277819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:47.278125 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:47.278280 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:47.278646 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 10:10:47.278872 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 10:10:47.279294 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:47.264625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:33143 (local address 127.0.0.1:35946)
0924 10:10:47.264958 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:47.264979 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:47.265117 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:47.265506 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:47.265510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:47.266071 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:47.266240 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:47.266249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.267411 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.267416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:47.268765 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:47.268781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:47.269938 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:47.269945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:47.270148 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:47.270887 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:47.270912 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:47.273463 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:47.276714 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:47.276722 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:47.276727 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:47.277133 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:47.277529 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:47.277534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:47.277538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:47.277695 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:47.278253 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:47.278258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:47.278410 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 10:10:47.278552 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 10:10:47.278856 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260924 10:10:47.284276 19740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31: Bootstrap starting.
I20260924 10:10:47.288000 19740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:47.289456 19740 log.cc:824] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:47.293893 19740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31: No bootstrap required, opened a new log
I20260924 10:10:47.301932 19740 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } }
I20260924 10:10:47.302290 19740 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:47.302415 19740 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb0a90d3a0443048315ce762c753c31, State: Initialized, Role: FOLLOWER
I20260924 10:10:47.303126 19740 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } }
I20260924 10:10:47.306937 19842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:10:47.307286 19842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } }
I20260924 10:10:47.308595 19842 leader_election.cc:288] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebb0a90d3a0443048315ce762c753c31 (127.6.2.253:34385), 24d71128e20e41c7a5357575c010859e (127.6.2.252:33143)
I20260924 10:10:47.309613 19789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "147cc9732da04f84a7e599389559cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24d71128e20e41c7a5357575c010859e" is_pre_election: true
I20260924 10:10:47.310153 19789 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 147cc9732da04f84a7e599389559cf49 in term 0.
I20260924 10:10:47.310851 19616 leader_election.cc:302] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 147cc9732da04f84a7e599389559cf49, 24d71128e20e41c7a5357575c010859e; no voters:
I20260924 10:10:47.311272 19842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:10:47.311417 19842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:10:47.311532 19842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:47.314329 19842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } }
I20260924 10:10:47.315249 19842 leader_election.cc:288] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [CANDIDATE]: Term 1 election: Requested vote from peers ebb0a90d3a0443048315ce762c753c31 (127.6.2.253:34385), 24d71128e20e41c7a5357575c010859e (127.6.2.252:33143)
I20260924 10:10:47.315498 19789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "147cc9732da04f84a7e599389559cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24d71128e20e41c7a5357575c010859e"
I20260924 10:10:47.315721 19789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:47.318185 19789 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 147cc9732da04f84a7e599389559cf49 in term 1.
I20260924 10:10:47.318588 19616 leader_election.cc:302] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 147cc9732da04f84a7e599389559cf49, 24d71128e20e41c7a5357575c010859e; no voters:
I20260924 10:10:47.318941 19842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:10:47.319159 19740 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:10:47.319197 19715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "147cc9732da04f84a7e599389559cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb0a90d3a0443048315ce762c753c31" is_pre_election: true
I20260924 10:10:47.321317 19715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 147cc9732da04f84a7e599389559cf49 in term 0.
I20260924 10:10:47.319348 19714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "147cc9732da04f84a7e599389559cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb0a90d3a0443048315ce762c753c31"
I20260924 10:10:47.321938 19714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:47.326053 19714 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 147cc9732da04f84a7e599389559cf49 in term 1.
I20260924 10:10:47.304343 19856 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } } }
I20260924 10:10:47.329541 19856 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:10:47.331315 19842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [term 1 LEADER]: Becoming Leader. State: Replica: 147cc9732da04f84a7e599389559cf49, State: Running, Role: LEADER
I20260924 10:10:47.332154 19842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } }
I20260924 10:10:47.344610 19857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 147cc9732da04f84a7e599389559cf49. Latest consensus state: current_term: 1 leader_uuid: "147cc9732da04f84a7e599389559cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } } }
I20260924 10:10:47.345132 19857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:47.364782 19862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:10:47.371330 19862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260924 10:10:47.379573 19870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:10:47.379840 19870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:10:47.388288 19789 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [term 1 FOLLOWER]: Refusing update from remote peer 147cc9732da04f84a7e599389559cf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:10:47.389329 19857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:10:47.396581 19714 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [term 1 FOLLOWER]: Refusing update from remote peer 147cc9732da04f84a7e599389559cf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:10:47.397465 19857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:10:47.421958 19827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 147cc9732da04f84a7e599389559cf49. Latest consensus state: current_term: 1 leader_uuid: "147cc9732da04f84a7e599389559cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } } }
I20260924 10:10:47.422325 19827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:10:47.435679 19842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "147cc9732da04f84a7e599389559cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } } }
I20260924 10:10:47.436086 19842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:47.436340 19872 mvcc.cc:205] Tried to move back new op lower bound from 7332842075690078208 to 7332842075486203904. Current Snapshot: MvccSnapshot[applied={T|T < 7332842075690078208}]
I20260924 10:10:47.439493 19827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "147cc9732da04f84a7e599389559cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } } }
I20260924 10:10:47.439802 19827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:10:47.440999 19862 catalog_manager.cc:1347] Generated new cluster ID: 62c4b090cbec4b1ba04dd0ef99917603
I20260924 10:10:47.441102 19862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:10:47.442629 19856 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 147cc9732da04f84a7e599389559cf49. Latest consensus state: current_term: 1 leader_uuid: "147cc9732da04f84a7e599389559cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } } }
I20260924 10:10:47.442939 19856 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:10:47.460770 19874 mvcc.cc:205] Tried to move back new op lower bound from 7332842075690078208 to 7332842075486203904. Current Snapshot: MvccSnapshot[applied={T|T < 7332842075690078208}]
I20260924 10:10:47.464388 19862 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:10:47.466610 19862 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:10:47.475757 19842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "147cc9732da04f84a7e599389559cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } } }
I20260924 10:10:47.476207 19842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49 [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:47.491344 19856 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "147cc9732da04f84a7e599389559cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cc9732da04f84a7e599389559cf49" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34039 } } peers { permanent_uuid: "ebb0a90d3a0443048315ce762c753c31" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 34385 } } peers { permanent_uuid: "24d71128e20e41c7a5357575c010859e" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 33143 } } }
I20260924 10:10:47.491616 19856 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:10:47.507314 19862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 147cc9732da04f84a7e599389559cf49: Generated new TSK 0
I20260924 10:10:47.508497 19862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:10:48.181466 19839 catalog_manager.cc:1259] Loaded cluster ID: 62c4b090cbec4b1ba04dd0ef99917603
I20260924 10:10:48.181568 19839 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e: loading cluster ID for follower catalog manager: success
I20260924 10:10:48.184784 19839 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e: acquiring CA information for follower catalog manager: success
I20260924 10:10:48.187816 19839 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 24d71128e20e41c7a5357575c010859e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:10:48.382162 19870 catalog_manager.cc:1259] Loaded cluster ID: 62c4b090cbec4b1ba04dd0ef99917603
I20260924 10:10:48.382279 19870 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31: loading cluster ID for follower catalog manager: success
I20260924 10:10:48.384856 19870 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31: acquiring CA information for follower catalog manager: success
I20260924 10:10:48.386731 19870 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ebb0a90d3a0443048315ce762c753c31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:10:48.643903 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 19599
I20260924 10:10:48.645658 19661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:48.761354 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 19599
W20260924 10:10:48.783425 19691 connection.cc:535] server connection from 127.0.0.1:60740 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:10:48.783746 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 19673
I20260924 10:10:48.784911 19735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:48.893500 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 19673
I20260924 10:10:48.914708 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 19747
I20260924 10:10:48.915985 19809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:49.028658 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 19747
2026-09-24T10:10:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4805 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:10:49 dist-test-slave-s3n3 krb5kdc[19891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:10:49 dist-test-slave-s3n3 krb5kdc[19891](info): set up 2 sockets
Sep 24 10:10:49 dist-test-slave-s3n3 krb5kdc[19891](info): commencing operation
krb5kdc: starting...
W20260924 10:10:50.147718 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.008s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:10:50 dist-test-slave-s3n3 krb5kdc[19891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:10:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:10:50.391546 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:35215
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:35215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:10:50.729946 19904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:50.730306 19904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:50.730476 19904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:50.737922 19904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:10:50.738077 19904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:50.738190 19904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:50.738297 19904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:10:50.738381 19904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:10:50.748430 19904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:35215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:35215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:50.749883 19904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:50.751242 19904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:50.763394 19909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:50.763324 19912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:50.763356 19910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:50.764577 19904 server_base.cc:1034] running on GCE node
I20260924 10:10:50.765455 19904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:50.767009 19904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:50.768294 19904 hybrid_clock.cc:648] HybridClock initialized: now 1790244650768259 us; error 32 us; skew 500 ppm
I20260924 10:10:50.768692 19904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:50 dist-test-slave-s3n3 krb5kdc[19891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244650, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:50.771965 19904 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:10:50.773900 19904 webserver.cc:460] Webserver started at http://127.6.2.254:38097/ using document root <none> and password file <none>
I20260924 10:10:50.774622 19904 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:50.774763 19904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:50.775148 19904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:10:50.778195 19904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "8d3bdd024b584c34ba4fe77e7f28e65e"
format_stamp: "Formatted at 2026-09-24 10:10:50 on dist-test-slave-s3n3"
server_key: "d5d1e90bfa939e50862f905fa50b7e34"
server_key_iv: "298da9a50ebc6206f758128e1171e242"
server_key_version: "encryptionkey@0"
I20260924 10:10:50.778986 19904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "8d3bdd024b584c34ba4fe77e7f28e65e"
format_stamp: "Formatted at 2026-09-24 10:10:50 on dist-test-slave-s3n3"
server_key: "d5d1e90bfa939e50862f905fa50b7e34"
server_key_iv: "298da9a50ebc6206f758128e1171e242"
server_key_version: "encryptionkey@0"
I20260924 10:10:50.784233 19904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:10:50.787940 19919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:50.789767 19904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:10:50.789984 19904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "8d3bdd024b584c34ba4fe77e7f28e65e"
format_stamp: "Formatted at 2026-09-24 10:10:50 on dist-test-slave-s3n3"
server_key: "d5d1e90bfa939e50862f905fa50b7e34"
server_key_iv: "298da9a50ebc6206f758128e1171e242"
server_key_version: "encryptionkey@0"
I20260924 10:10:50.790215 19904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:50.827867 19904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:50.832298 19904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:50.832631 19904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:50.852839 19904 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:35215
I20260924 10:10:50.852926 19970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:35215 every 8 connection(s)
I20260924 10:10:50.854255 19904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:10:50.856606 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 19904
I20260924 10:10:50.856832 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:10:50.857515 6155 external_mini_cluster.cc:1428] Setting key fffbc321d0b9b47aac05ba758f21541e
I20260924 10:10:50.858328 19971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:50.872648 19971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e: Bootstrap starting.
Sep 24 10:10:50 dist-test-slave-s3n3 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:50.876111 19971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:50.877615 19971 log.cc:824] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:50.881022 19971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e: No bootstrap required, opened a new log
I20260924 10:10:50.884230 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:50.859539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.6.2.254:35215)
0924 10:10:50.860404 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:50.860420 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:50.860467 (+ 47us) server_negotiation.cc:407] Connection header received
0924 10:10:50.863042 (+ 2575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:50.863093 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:50.863635 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:50.864094 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:50.865616 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:50.867577 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:50.869404 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:50.870555 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:50.870756 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:50.875364 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:50.875377 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:50.875397 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:50.875445 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:50.879470 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:50.880246 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:50.880263 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:50.880270 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:50.880482 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:50.880896 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:50.880901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:50.880904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:50.881617 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:50.881937 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:50.882309 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 10:10:50.882741 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":375,"threads_started":1}
I20260924 10:10:50.888367 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:10:50.890458 19971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3bdd024b584c34ba4fe77e7f28e65e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35215 } }
I20260924 10:10:50.890945 19971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:50.891108 19971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3bdd024b584c34ba4fe77e7f28e65e, State: Initialized, Role: FOLLOWER
I20260924 10:10:50.891733 19971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3bdd024b584c34ba4fe77e7f28e65e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35215 } }
I20260924 10:10:50.891956 19971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:10:50.892107 19971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:10:50.892259 19971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:50.894606 19971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3bdd024b584c34ba4fe77e7f28e65e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35215 } }
I20260924 10:10:50.895229 19971 leader_election.cc:302] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d3bdd024b584c34ba4fe77e7f28e65e; no voters:
I20260924 10:10:50.895807 19971 leader_election.cc:288] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:10:50.895974 19976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:10:50.896701 19976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [term 1 LEADER]: Becoming Leader. State: Replica: 8d3bdd024b584c34ba4fe77e7f28e65e, State: Running, Role: LEADER
I20260924 10:10:50.897307 19976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3bdd024b584c34ba4fe77e7f28e65e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35215 } }
I20260924 10:10:50.897969 19971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:10:50.900099 19978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d3bdd024b584c34ba4fe77e7f28e65e. Latest consensus state: current_term: 1 leader_uuid: "8d3bdd024b584c34ba4fe77e7f28e65e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3bdd024b584c34ba4fe77e7f28e65e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35215 } } }
I20260924 10:10:50.900127 19977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d3bdd024b584c34ba4fe77e7f28e65e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3bdd024b584c34ba4fe77e7f28e65e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 35215 } } }
I20260924 10:10:50.900530 19977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:50.900534 19978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:50.902669 19982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:10:50.908226 19982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:10:50.917009 19982 catalog_manager.cc:1347] Generated new cluster ID: 457aadcbdf3d4b1dad4437e06fcaf647
I20260924 10:10:50.917124 19982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:10:50.938685 19982 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:10:50.940146 19982 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:10:50.946851 19982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d3bdd024b584c34ba4fe77e7f28e65e: Generated new TSK 0
I20260924 10:10:50.947734 19982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:10:51.001543 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 19904
I20260924 10:10:51.002897 19966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:51.104043 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 19904
2026-09-24T10:10:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2070 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:10:51 dist-test-slave-s3n3 krb5kdc[19999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:10:51 dist-test-slave-s3n3 krb5kdc[19999](info): set up 2 sockets
Sep 24 10:10:51 dist-test-slave-s3n3 krb5kdc[19999](info): commencing operation
krb5kdc: starting...
W20260924 10:10:52.231541 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:10:52 dist-test-slave-s3n3 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:10:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:10:52.479952 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36627
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:36627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:10:52.821730 20012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:52.822062 20012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:52.822223 20012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:52.829782 20012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:10:52.829919 20012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:52.830010 20012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:52.830106 20012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:10:52.830180 20012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:10:52.840606 20012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:36627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:52.842115 20012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:52.843474 20012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:52.856489 20017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:52.856527 20018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:52.856565 20020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:52.857204 20012 server_base.cc:1034] running on GCE node
I20260924 10:10:52.858089 20012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:52.859757 20012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:52.860987 20012 hybrid_clock.cc:648] HybridClock initialized: now 1790244652860944 us; error 34 us; skew 500 ppm
I20260924 10:10:52.861310 20012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:52 dist-test-slave-s3n3 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:52.871785 20012 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:10:52.873548 20012 webserver.cc:460] Webserver started at http://127.6.2.254:34357/ using document root <none> and password file <none>
I20260924 10:10:52.874317 20012 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:52.874428 20012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:52.874706 20012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:10:52.878147 20012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "ddf119c5899e4b3080bdf81412d3f7d9"
format_stamp: "Formatted at 2026-09-24 10:10:52 on dist-test-slave-s3n3"
server_key: "a5f735ed2db9197c6df3c754805fd63a"
server_key_iv: "2cca7f5224c9f56c240dd4a51efd056d"
server_key_version: "encryptionkey@0"
I20260924 10:10:52.878960 20012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "ddf119c5899e4b3080bdf81412d3f7d9"
format_stamp: "Formatted at 2026-09-24 10:10:52 on dist-test-slave-s3n3"
server_key: "a5f735ed2db9197c6df3c754805fd63a"
server_key_iv: "2cca7f5224c9f56c240dd4a51efd056d"
server_key_version: "encryptionkey@0"
I20260924 10:10:52.884795 20012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 10:10:52.888597 20027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:52.890123 20012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:10:52.890286 20012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "ddf119c5899e4b3080bdf81412d3f7d9"
format_stamp: "Formatted at 2026-09-24 10:10:52 on dist-test-slave-s3n3"
server_key: "a5f735ed2db9197c6df3c754805fd63a"
server_key_iv: "2cca7f5224c9f56c240dd4a51efd056d"
server_key_version: "encryptionkey@0"
I20260924 10:10:52.890476 20012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:52.908469 20012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:52.913475 20012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:52.913748 20012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:52.933820 20012 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:36627
I20260924 10:10:52.933880 20078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:36627 every 8 connection(s)
I20260924 10:10:52.935305 20012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:10:52.939370 20079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:52.945120 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 20012
I20260924 10:10:52.945358 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:10:52.946308 6155 external_mini_cluster.cc:1428] Setting key 8fdd1fc70793335647d9ed7eaa75fc10
I20260924 10:10:52.951903 20079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Bootstrap starting.
I20260924 10:10:52.955204 20079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:52.956305 20079 log.cc:824] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:52.959755 20079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: No bootstrap required, opened a new log
Sep 24 10:10:52 dist-test-slave-s3n3 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:52.969204 20079 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } }
I20260924 10:10:52.969738 20079 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:52.969877 20079 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf119c5899e4b3080bdf81412d3f7d9, State: Initialized, Role: FOLLOWER
I20260924 10:10:52.970549 20079 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } }
I20260924 10:10:52.970804 20079 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:10:52.970986 20079 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:10:52.971235 20079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:52.972755 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:52.949174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.6.2.254:36627)
0924 10:10:52.950284 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:52.950298 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:52.950347 (+ 49us) server_negotiation.cc:407] Connection header received
0924 10:10:52.952329 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:52.952353 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:52.952782 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:52.953206 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:52.955027 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:52.956704 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:52.958566 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:52.959339 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:52.959478 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:52.964585 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:52.964593 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:52.964615 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:52.964662 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:52.968395 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:52.969138 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:52.969145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:52.969152 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:52.969362 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:52.969786 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:52.969792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:52.969795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:52.970425 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:52.970755 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:52.971233 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 10:10:52.971688 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":429,"threads_started":1}
I20260924 10:10:52.974215 20079 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } }
I20260924 10:10:52.974778 20079 leader_election.cc:302] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf119c5899e4b3080bdf81412d3f7d9; no voters:
I20260924 10:10:52.975606 20079 leader_election.cc:288] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:10:52.975716 20084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:10:52.976725 20084 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 1 LEADER]: Becoming Leader. State: Replica: ddf119c5899e4b3080bdf81412d3f7d9, State: Running, Role: LEADER
I20260924 10:10:52.977311 20084 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } }
I20260924 10:10:52.978133 20079 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:10:52.980275 20085 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } } }
I20260924 10:10:52.980644 20085 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:52.983352 20086 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf119c5899e4b3080bdf81412d3f7d9. Latest consensus state: current_term: 1 leader_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } } }
I20260924 10:10:52.983843 20086 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:52.984947 20089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:10:52.990078 20089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:10:53.000407 20089 catalog_manager.cc:1347] Generated new cluster ID: 66b602989e99440ca5b1e95c573f3fe7
I20260924 10:10:53.000538 20089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:10:53.020905 20089 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:10:53.022344 20089 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:10:53.029083 20089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Generated new TSK 0
I20260924 10:10:53.029946 20089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:10:53.087939 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36627
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:10:53.429589 20105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:53.429889 20105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:53.430042 20105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:53.438115 20105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:53.438232 20105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:53.438410 20105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:10:53.448635 20105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:53.449993 20105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:53.451524 20105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:53.464807 20113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:53.464848 20110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:53.465235 20111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:53.465854 20105 server_base.cc:1034] running on GCE node
I20260924 10:10:53.466452 20105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:53.467466 20105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:53.468662 20105 hybrid_clock.cc:648] HybridClock initialized: now 1790244653468597 us; error 55 us; skew 500 ppm
I20260924 10:10:53.469002 20105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:53 dist-test-slave-s3n3 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244653, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:53.472675 20105 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:10:53.475415 20105 webserver.cc:460] Webserver started at http://127.6.2.193:36011/ using document root <none> and password file <none>
I20260924 10:10:53.476137 20105 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:53.476282 20105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:53.476588 20105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:10:53.479817 20105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "df0b97be3cae49b8985fd641f6375f87"
format_stamp: "Formatted at 2026-09-24 10:10:53 on dist-test-slave-s3n3"
server_key: "28fee85218e345f36a31c71d829ea65b"
server_key_iv: "584ca27498ffec8b40a8c72e1fb3419c"
server_key_version: "encryptionkey@0"
I20260924 10:10:53.480599 20105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "df0b97be3cae49b8985fd641f6375f87"
format_stamp: "Formatted at 2026-09-24 10:10:53 on dist-test-slave-s3n3"
server_key: "28fee85218e345f36a31c71d829ea65b"
server_key_iv: "584ca27498ffec8b40a8c72e1fb3419c"
server_key_version: "encryptionkey@0"
I20260924 10:10:53.486130 20105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:10:53.489683 20120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:53.491004 20105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:10:53.491257 20105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "df0b97be3cae49b8985fd641f6375f87"
format_stamp: "Formatted at 2026-09-24 10:10:53 on dist-test-slave-s3n3"
server_key: "28fee85218e345f36a31c71d829ea65b"
server_key_iv: "584ca27498ffec8b40a8c72e1fb3419c"
server_key_version: "encryptionkey@0"
I20260924 10:10:53.491477 20105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:53.506317 20105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:53.510772 20105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:53.511133 20105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:53.512125 20105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:10:53.513545 20105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:10:53.513618 20105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:53.513743 20105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:10:53.513813 20105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:53.548329 20105 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:37927
I20260924 10:10:53.548372 20232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:37927 every 8 connection(s)
I20260924 10:10:53.549650 20105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:10:53.553373 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 20105
I20260924 10:10:53.553587 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:10:53.554296 6155 external_mini_cluster.cc:1428] Setting key 02d4c27832c96fd9401bed37a8b48c71
Sep 24 10:10:53 dist-test-slave-s3n3 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244653, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:53.571972 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:53.552333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:35905 (local address 127.6.2.254:36627)
0924 10:10:53.552739 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:53.552744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:53.553859 (+ 1115us) server_negotiation.cc:407] Connection header received
0924 10:10:53.556174 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:53.556180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:53.556274 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:53.556362 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:53.557923 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:53.559121 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:53.560693 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:53.561467 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:53.561611 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:53.565451 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:53.565457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:53.565460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:53.565494 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:53.568126 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:53.568903 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:53.568909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:53.568912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:53.569042 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:53.569621 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:53.569629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:53.569635 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:53.570060 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:53.570229 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:53.571165 (+ 936us) server_negotiation.cc:299] Negotiation successful
0924 10:10:53.571443 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":192,"threads_started":1}
I20260924 10:10:53.572911 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:53.552861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:35905)
0924 10:10:53.553677 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:53.553721 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:53.555873 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:53.556636 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:53.556644 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:53.557153 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:53.557728 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:53.557746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:53.559296 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:53.559300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:53.560530 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:53.560542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:53.561676 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:53.561681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:53.561889 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:53.562601 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:53.562623 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:53.565221 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:53.568280 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:53.568291 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:53.568308 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:53.568773 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:53.569156 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:53.569162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:53.569168 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:53.569460 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:53.570229 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:53.570236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:53.570647 (+ 411us) client_negotiation.cc:769] Sending connection context
0924 10:10:53.571009 (+ 362us) client_negotiation.cc:240] Negotiation successful
0924 10:10:53.571516 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":357,"threads_started":1}
I20260924 10:10:53.574568 20233 heartbeater.cc:347] Connected to a master server at 127.6.2.254:36627
I20260924 10:10:53.574951 20233 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:53.575980 20233 heartbeater.cc:511] Master 127.6.2.254:36627 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
I20260924 10:10:53.578576 20044 ts_manager.cc:194] Registered new tserver with Master: df0b97be3cae49b8985fd641f6375f87 (127.6.2.193:37927)
I20260924 10:10:53.580406 20044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:35905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:10:53.611852 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36627
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:10:53.946055 20239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:53.946321 20239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:53.946486 20239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:53.954092 20239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:53.954209 20239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:53.954385 20239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:10:53.964499 20239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:53.965883 20239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:53.967267 20239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:53.981354 20244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:53.981310 20247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:53.981276 20245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:53.982476 20239 server_base.cc:1034] running on GCE node
I20260924 10:10:53.983302 20239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:53.984460 20239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:53.985710 20239 hybrid_clock.cc:648] HybridClock initialized: now 1790244653985635 us; error 59 us; skew 500 ppm
I20260924 10:10:53.986070 20239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:53 dist-test-slave-s3n3 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244653, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:53.989431 20239 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:10:53.991024 20239 webserver.cc:460] Webserver started at http://127.6.2.194:45043/ using document root <none> and password file <none>
I20260924 10:10:53.991895 20239 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:53.992045 20239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:53.992367 20239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:10:53.995591 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "5689d5ac27e843f29a43c321eccaced6"
format_stamp: "Formatted at 2026-09-24 10:10:53 on dist-test-slave-s3n3"
server_key: "c205be34fe412fbcb2783337b9b2ca07"
server_key_iv: "b2bd0a33f04dd7c9c53eff6b3c41a7d5"
server_key_version: "encryptionkey@0"
I20260924 10:10:53.996421 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "5689d5ac27e843f29a43c321eccaced6"
format_stamp: "Formatted at 2026-09-24 10:10:53 on dist-test-slave-s3n3"
server_key: "c205be34fe412fbcb2783337b9b2ca07"
server_key_iv: "b2bd0a33f04dd7c9c53eff6b3c41a7d5"
server_key_version: "encryptionkey@0"
I20260924 10:10:54.001818 20239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 10:10:54.005714 20254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:54.007320 20239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:10:54.007511 20239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "5689d5ac27e843f29a43c321eccaced6"
format_stamp: "Formatted at 2026-09-24 10:10:53 on dist-test-slave-s3n3"
server_key: "c205be34fe412fbcb2783337b9b2ca07"
server_key_iv: "b2bd0a33f04dd7c9c53eff6b3c41a7d5"
server_key_version: "encryptionkey@0"
I20260924 10:10:54.007748 20239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:54.030406 20239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:54.035261 20239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:54.035580 20239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:54.036614 20239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:10:54.038089 20239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:10:54.038199 20239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:54.038352 20239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:10:54.038424 20239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:54.073843 20239 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:38077
I20260924 10:10:54.073948 20366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:38077 every 8 connection(s)
I20260924 10:10:54.075266 20239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:10:54.076560 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 20239
I20260924 10:10:54.076763 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:10:54.077224 6155 external_mini_cluster.cc:1428] Setting key e82f941ed46b05969852191d9398e02d
Sep 24 10:10:54 dist-test-slave-s3n3 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244653, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:54.097537 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:54.078077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:60183 (local address 127.6.2.254:36627)
0924 10:10:54.078542 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:54.078548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:54.079774 (+ 1226us) server_negotiation.cc:407] Connection header received
0924 10:10:54.081702 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:54.081707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:54.081812 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:54.081932 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:54.083530 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.084260 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.085822 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.086241 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.086393 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:54.090552 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:54.090558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:54.090562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:54.090597 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:54.093600 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:54.094755 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:54.094760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:54.094762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:54.094918 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:54.095512 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:54.095516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:54.095518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:54.095856 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:54.096033 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:54.096900 (+ 867us) server_negotiation.cc:299] Negotiation successful
0924 10:10:54.097139 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":197,"threads_started":1}
I20260924 10:10:54.098565 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:54.078689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:60183)
0924 10:10:54.079556 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:54.079601 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:54.081427 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:54.082187 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:54.082195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:54.082711 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:54.083350 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:54.083369 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.084433 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.084438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:54.085658 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:54.085670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.086480 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.086487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:54.086718 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:54.087513 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:54.087535 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:54.090250 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:54.093766 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:54.093779 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:54.093801 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:54.094612 (+ 811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:54.095047 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:54.095055 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:54.095079 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:54.095387 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:54.096006 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:54.096015 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:54.096510 (+ 495us) client_negotiation.cc:769] Sending connection context
0924 10:10:54.096795 (+ 285us) client_negotiation.cc:240] Negotiation successful
0924 10:10:54.097362 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":398,"threads_started":1}
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:10:54.100142 20367 heartbeater.cc:347] Connected to a master server at 127.6.2.254:36627
I20260924 10:10:54.100459 20367 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:54.101172 20367 heartbeater.cc:511] Master 127.6.2.254:36627 requested a full tablet report, sending...
I20260924 10:10:54.102660 20044 ts_manager.cc:194] Registered new tserver with Master: 5689d5ac27e843f29a43c321eccaced6 (127.6.2.194:38077)
I20260924 10:10:54.103523 20044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:60183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:10:54.136003 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36627
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:10:54.472371 20373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:54.472730 20373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:54.472924 20373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:54.480499 20373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:54.480609 20373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:54.480798 20373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:10:54.490679 20373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:54.492154 20373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:54.493511 20373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:54.507296 20379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:54.507385 20378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:54.507416 20381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:54.508337 20373 server_base.cc:1034] running on GCE node
I20260924 10:10:54.508963 20373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:54.510030 20373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:54.511185 20373 hybrid_clock.cc:648] HybridClock initialized: now 1790244654511145 us; error 34 us; skew 500 ppm
I20260924 10:10:54.511536 20373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:54 dist-test-slave-s3n3 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244654, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:54.514745 20373 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:10:54.516589 20373 webserver.cc:460] Webserver started at http://127.6.2.195:36277/ using document root <none> and password file <none>
I20260924 10:10:54.517264 20373 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:54.517359 20373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:54.517609 20373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:10:54.520745 20373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "22adf1fa898e47848eee442f6bf771c4"
format_stamp: "Formatted at 2026-09-24 10:10:54 on dist-test-slave-s3n3"
server_key: "940a40226933b0f01ce0aeeeeda7e4f1"
server_key_iv: "82ea5cacf9b5c713e2fdc3f7c1af7a93"
server_key_version: "encryptionkey@0"
I20260924 10:10:54.521463 20373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "22adf1fa898e47848eee442f6bf771c4"
format_stamp: "Formatted at 2026-09-24 10:10:54 on dist-test-slave-s3n3"
server_key: "940a40226933b0f01ce0aeeeeda7e4f1"
server_key_iv: "82ea5cacf9b5c713e2fdc3f7c1af7a93"
server_key_version: "encryptionkey@0"
I20260924 10:10:54.526721 20373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:10:54.530370 20388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:54.531879 20373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:10:54.532069 20373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "22adf1fa898e47848eee442f6bf771c4"
format_stamp: "Formatted at 2026-09-24 10:10:54 on dist-test-slave-s3n3"
server_key: "940a40226933b0f01ce0aeeeeda7e4f1"
server_key_iv: "82ea5cacf9b5c713e2fdc3f7c1af7a93"
server_key_version: "encryptionkey@0"
I20260924 10:10:54.532290 20373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:54.544909 20373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:54.550504 20373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:54.550916 20373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:54.552224 20373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:10:54.553591 20373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:10:54.553706 20373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:54.553843 20373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:10:54.553925 20373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:54.583693 20233 heartbeater.cc:503] Master 127.6.2.254:36627 was elected leader, sending a full tablet report...
I20260924 10:10:54.591566 20373 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:43271
I20260924 10:10:54.591614 20500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:43271 every 8 connection(s)
I20260924 10:10:54.593012 20373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:10:54.601213 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 20373
I20260924 10:10:54.601418 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:10:54.601950 6155 external_mini_cluster.cc:1428] Setting key be206a0843199ada36ca84c4c78dcedb
Sep 24 10:10:54 dist-test-slave-s3n3 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244654, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:54.614216 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:54.595747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:56071 (local address 127.6.2.254:36627)
0924 10:10:54.595999 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:54.596008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:54.597175 (+ 1167us) server_negotiation.cc:407] Connection header received
0924 10:10:54.598993 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:54.598998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:54.599143 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:54.599237 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:54.600878 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.602047 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.604153 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.604549 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.604686 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:54.608364 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:54.608370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:54.608373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:54.608407 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:54.611004 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:54.611860 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:54.611866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:54.611870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:54.611992 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:54.612413 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:54.612421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:54.612426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:54.612732 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:54.612921 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:54.613614 (+ 693us) server_negotiation.cc:299] Negotiation successful
0924 10:10:54.613868 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 10:10:54.615267 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:54.596263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:56071)
0924 10:10:54.597002 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:54.597052 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:54.598727 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:54.599496 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:54.599508 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:54.600126 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:54.600690 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:54.600708 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.602256 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.602261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:54.603996 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:54.604009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.604788 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.604796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:54.605011 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:54.605769 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:54.605792 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:54.608125 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:54.611201 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:54.611210 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:54.611228 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:54.611734 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:54.612106 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:54.612110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:54.612114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:54.612305 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:54.612894 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:54.612902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:54.613249 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 10:10:54.613522 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 10:10:54.614104 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":296,"threads_started":1}
I20260924 10:10:54.616922 20501 heartbeater.cc:347] Connected to a master server at 127.6.2.254:36627
I20260924 10:10:54.617297 20501 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:54.618134 20501 heartbeater.cc:511] Master 127.6.2.254:36627 requested a full tablet report, sending...
I20260924 10:10:54.619568 20043 ts_manager.cc:194] Registered new tserver with Master: 22adf1fa898e47848eee442f6bf771c4 (127.6.2.195:43271)
I20260924 10:10:54.620400 20043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:56071
I20260924 10:10:54.631286 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:10:54.649647 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:54.636483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.6.2.254:36627)
0924 10:10:54.636770 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:54.636787 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:54.637116 (+ 329us) server_negotiation.cc:407] Connection header received
0924 10:10:54.637319 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:54.637324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:54.637425 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:54.637522 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:54.638945 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.639726 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.641058 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.641523 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.641621 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:54.644198 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:54.644203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:54.644206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:54.644240 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:54.646985 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:54.647745 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:54.647749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:54.647751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:54.647917 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:54.648248 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:54.648251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:54.648253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:54.648599 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:54.648825 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:54.649058 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 10:10:54.649258 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 10:10:54.654520 20043 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:10:54.662384 20043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:10:54.698675 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:54.675054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.6.2.195:43271)
0924 10:10:54.689371 (+ 14317us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:54.689381 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:54.689419 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:10:54.689524 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:54.689533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:54.689810 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:54.690075 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:54.691349 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.693005 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.696020 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.697457 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.697622 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:54.697894 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:54.698168 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 10:10:54.698328 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14094,"thread_start_us":15213,"threads_started":1}
I20260924 10:10:54.698736 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:54.674809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:43271 (local address 127.0.0.1:35474)
0924 10:10:54.688285 (+ 13476us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:54.688366 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:54.688665 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:54.690576 (+ 1911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:54.690587 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:54.690626 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:54.691133 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:54.691149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.693182 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.693188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:54.695803 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:54.695819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.697684 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.697690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:54.697902 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:54.697971 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 10:10:54.698106 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 10:10:54.698305 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12755,"mutex_wait_us":21,"thread_start_us":14348,"threads_started":1}
I20260924 10:10:54.700340 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:54.675134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:37927 (local address 127.0.0.1:47438)
0924 10:10:54.688945 (+ 13811us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:54.688966 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:54.689086 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:54.691925 (+ 2839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:54.691930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:54.691961 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:54.692381 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:54.692391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.694378 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.694385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:54.697328 (+ 2943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:54.697342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.699453 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.699459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:54.699634 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:54.699671 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 10:10:54.699768 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 10:10:54.699926 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13643,"mutex_wait_us":1,"thread_start_us":11969,"threads_started":1}
I20260924 10:10:54.700372 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:54.675253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47438 (local address 127.6.2.193:37927)
0924 10:10:54.691083 (+ 15830us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:54.691094 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:54.691127 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:10:54.691277 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:54.691290 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:54.691620 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:54.691820 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:54.692618 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.694198 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.697521 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.699045 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.699198 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:54.699467 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:54.699840 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 10:10:54.700008 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15607,"thread_start_us":15442,"threads_started":1}
I20260924 10:10:54.702957 20436 tablet_service.cc:1467] Processing CreateTablet for tablet 83da9e288c174ab39c7514fe485db167 (DEFAULT_TABLE table=test-table [id=2d90a818f51c4f0d90f1e241fed04914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:10:54.703631 20168 tablet_service.cc:1467] Processing CreateTablet for tablet 83da9e288c174ab39c7514fe485db167 (DEFAULT_TABLE table=test-table [id=2d90a818f51c4f0d90f1e241fed04914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:10:54.704270 20436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83da9e288c174ab39c7514fe485db167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:54.704716 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:54.677223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:38077 (local address 127.0.0.1:47264)
0924 10:10:54.691934 (+ 14711us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:54.691951 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:54.692081 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:54.692909 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:54.692913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:54.692941 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:54.693441 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:54.693455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.695344 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.695350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:54.697678 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:54.697693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.699727 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.699760 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:54.699983 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:54.700015 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:10:54.700107 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 10:10:54.704332 (+ 4225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14550,"mutex_wait_us":29,"thread_start_us":14399,"threads_started":1}
I20260924 10:10:54.704741 20168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83da9e288c174ab39c7514fe485db167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:54.705626 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:54.677942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.6.2.194:38077)
0924 10:10:54.691426 (+ 13484us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:54.691438 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:54.692052 (+ 614us) server_negotiation.cc:407] Connection header received
0924 10:10:54.692256 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:54.692264 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:54.692510 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:54.692677 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:54.693655 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.695211 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.697870 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:54.699282 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:54.699417 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:54.699602 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:54.705109 (+ 5507us) server_negotiation.cc:299] Negotiation successful
0924 10:10:54.705316 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13313,"thread_start_us":13166,"threads_started":1}
I20260924 10:10:54.709106 20302 tablet_service.cc:1467] Processing CreateTablet for tablet 83da9e288c174ab39c7514fe485db167 (DEFAULT_TABLE table=test-table [id=2d90a818f51c4f0d90f1e241fed04914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:10:54.710490 20302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83da9e288c174ab39c7514fe485db167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:54.718992 20519 tablet_bootstrap.cc:492] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87: Bootstrap starting.
I20260924 10:10:54.721776 20520 tablet_bootstrap.cc:492] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4: Bootstrap starting.
I20260924 10:10:54.723965 20519 tablet_bootstrap.cc:654] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:54.725279 20519 log.cc:824] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:54.725327 20521 tablet_bootstrap.cc:492] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6: Bootstrap starting.
I20260924 10:10:54.725468 20520 tablet_bootstrap.cc:654] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:54.726738 20520 log.cc:824] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:54.728780 20519 tablet_bootstrap.cc:492] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87: No bootstrap required, opened a new log
I20260924 10:10:54.728965 20521 tablet_bootstrap.cc:654] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:54.729198 20519 ts_tablet_manager.cc:1397] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260924 10:10:54.729805 20520 tablet_bootstrap.cc:492] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4: No bootstrap required, opened a new log
I20260924 10:10:54.730078 20520 ts_tablet_manager.cc:1397] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260924 10:10:54.730137 20521 log.cc:824] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:54.732569 20521 tablet_bootstrap.cc:492] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6: No bootstrap required, opened a new log
I20260924 10:10:54.732914 20521 ts_tablet_manager.cc:1397] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260924 10:10:54.740473 20519 raft_consensus.cc:348] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } } peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } }
I20260924 10:10:54.740995 20519 raft_consensus.cc:374] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:54.741097 20519 raft_consensus.cc:729] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0b97be3cae49b8985fd641f6375f87, State: Initialized, Role: FOLLOWER
I20260924 10:10:54.740869 20520 raft_consensus.cc:348] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } } peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } }
I20260924 10:10:54.740924 20521 raft_consensus.cc:348] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } } peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } }
I20260924 10:10:54.741299 20520 raft_consensus.cc:374] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:54.741366 20521 raft_consensus.cc:374] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:54.741407 20520 raft_consensus.cc:729] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22adf1fa898e47848eee442f6bf771c4, State: Initialized, Role: FOLLOWER
I20260924 10:10:54.741468 20521 raft_consensus.cc:729] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5689d5ac27e843f29a43c321eccaced6, State: Initialized, Role: FOLLOWER
I20260924 10:10:54.742041 20521 consensus_queue.cc:260] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } } peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } }
I20260924 10:10:54.742003 20519 consensus_queue.cc:260] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } } peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } }
I20260924 10:10:54.742141 20520 consensus_queue.cc:260] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } } peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } }
I20260924 10:10:54.743136 20367 heartbeater.cc:503] Master 127.6.2.254:36627 was elected leader, sending a full tablet report...
I20260924 10:10:54.743893 20521 ts_tablet_manager.cc:1428] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260924 10:10:54.746449 20501 heartbeater.cc:503] Master 127.6.2.254:36627 was elected leader, sending a full tablet report...
I20260924 10:10:54.746718 20519 ts_tablet_manager.cc:1428] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87: Time spent starting tablet: real 0.016s user 0.004s sys 0.011s
I20260924 10:10:54.747927 20520 ts_tablet_manager.cc:1428] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4: Time spent starting tablet: real 0.018s user 0.005s sys 0.013s
W20260924 10:10:54.802109 20234 tablet.cc:2369] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:10:54.827416 20368 tablet.cc:2369] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:10:54.844754 20502 tablet.cc:2369] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:10:55.192575 20527 raft_consensus.cc:482] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:10:55.192906 20527 raft_consensus.cc:504] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } } peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } }
I20260924 10:10:55.194563 20527 leader_election.cc:288] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0b97be3cae49b8985fd641f6375f87 (127.6.2.193:37927), 5689d5ac27e843f29a43c321eccaced6 (127.6.2.194:38077)
I20260924 10:10:55.203716 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:55.195001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:45525 (local address 127.6.2.194:38077)
0924 10:10:55.195237 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:55.195244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:55.195866 (+ 622us) server_negotiation.cc:407] Connection header received
0924 10:10:55.196091 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:55.196097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:55.196177 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:55.196284 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:55.197184 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.198257 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.201283 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.202543 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.202703 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:55.202808 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:55.203317 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 10:10:55.203456 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 10:10:55.203718 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:55.195135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:54181 (local address 127.6.2.193:37927)
0924 10:10:55.195441 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:55.195451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:55.195917 (+ 466us) server_negotiation.cc:407] Connection header received
0924 10:10:55.196089 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:55.196094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:55.196176 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:55.196284 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:55.197049 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.198239 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.201288 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.202567 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.202703 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:55.202811 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:55.203317 (+ 506us) server_negotiation.cc:299] Negotiation successful
0924 10:10:55.203464 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:10:55.203840 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:55.194936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:37927 (local address 127.6.2.195:54181)
0924 10:10:55.195730 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:55.195757 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:55.195914 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:55.196320 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:55.196326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:55.196427 (+ 101us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:55.196854 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:55.196873 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.198407 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.198410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:55.201125 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:55.201133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.202767 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.202773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:55.203044 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:55.203175 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 10:10:55.203273 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 10:10:55.203431 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":279,"spinlock_wait_cycles":896,"thread_start_us":545,"threads_started":1}
I20260924 10:10:55.204308 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:55.194864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:38077 (local address 127.6.2.195:45525)
0924 10:10:55.195730 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:55.195752 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:55.195949 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:55.196400 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:55.196404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:55.196425 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:55.196999 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:55.197011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.198400 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.198405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:55.201114 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:55.201130 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.202767 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.202773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:55.203051 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:55.203174 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 10:10:55.203273 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 10:10:55.203642 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":278,"spinlock_wait_cycles":20864,"thread_start_us":550,"threads_started":1}
I20260924 10:10:55.205037 20188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83da9e288c174ab39c7514fe485db167" candidate_uuid: "22adf1fa898e47848eee442f6bf771c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0b97be3cae49b8985fd641f6375f87" is_pre_election: true
I20260924 10:10:55.205555 20188 raft_consensus.cc:2454] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22adf1fa898e47848eee442f6bf771c4 in term 0.
I20260924 10:10:55.206034 20322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83da9e288c174ab39c7514fe485db167" candidate_uuid: "22adf1fa898e47848eee442f6bf771c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5689d5ac27e843f29a43c321eccaced6" is_pre_election: true
I20260924 10:10:55.206629 20391 leader_election.cc:302] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22adf1fa898e47848eee442f6bf771c4, df0b97be3cae49b8985fd641f6375f87; no voters:
I20260924 10:10:55.206854 20322 raft_consensus.cc:2454] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22adf1fa898e47848eee442f6bf771c4 in term 0.
I20260924 10:10:55.207103 20527 raft_consensus.cc:2781] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:10:55.207304 20527 raft_consensus.cc:482] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:10:55.207417 20527 raft_consensus.cc:3037] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:55.209829 20527 raft_consensus.cc:504] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } } peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } }
I20260924 10:10:55.210531 20527 leader_election.cc:288] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [CANDIDATE]: Term 1 election: Requested vote from peers df0b97be3cae49b8985fd641f6375f87 (127.6.2.193:37927), 5689d5ac27e843f29a43c321eccaced6 (127.6.2.194:38077)
I20260924 10:10:55.211136 20188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83da9e288c174ab39c7514fe485db167" candidate_uuid: "22adf1fa898e47848eee442f6bf771c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0b97be3cae49b8985fd641f6375f87"
I20260924 10:10:55.211386 20188 raft_consensus.cc:3037] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:55.211400 20322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83da9e288c174ab39c7514fe485db167" candidate_uuid: "22adf1fa898e47848eee442f6bf771c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5689d5ac27e843f29a43c321eccaced6"
I20260924 10:10:55.211712 20322 raft_consensus.cc:3037] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:55.213896 20188 raft_consensus.cc:2454] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22adf1fa898e47848eee442f6bf771c4 in term 1.
I20260924 10:10:55.214315 20322 raft_consensus.cc:2454] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22adf1fa898e47848eee442f6bf771c4 in term 1.
I20260924 10:10:55.214475 20391 leader_election.cc:302] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22adf1fa898e47848eee442f6bf771c4, df0b97be3cae49b8985fd641f6375f87; no voters:
I20260924 10:10:55.214854 20527 raft_consensus.cc:2781] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:10:55.215451 20527 raft_consensus.cc:686] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [term 1 LEADER]: Becoming Leader. State: Replica: 22adf1fa898e47848eee442f6bf771c4, State: Running, Role: LEADER
I20260924 10:10:55.216032 20527 consensus_queue.cc:237] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } } peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } }
I20260924 10:10:55.221354 20044 catalog_manager.cc:5495] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22adf1fa898e47848eee442f6bf771c4 (127.6.2.195). New cstate: current_term: 1 leader_uuid: "22adf1fa898e47848eee442f6bf771c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } health_report { overall_health: UNKNOWN } } }
I20260924 10:10:55.268054 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:55.261915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.6.2.195:43271)
0924 10:10:55.262584 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:55.262590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:55.262609 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:10:55.262751 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:55.262755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:55.262841 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:55.263013 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:10:55.263466 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.264163 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.265743 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.266145 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.266286 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:55.266789 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:10:55.267404 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:10:55.267563 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:55.267659 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 10:10:55.267745 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":407,"threads_started":1}
I20260924 10:10:55.276638 20322 raft_consensus.cc:1261] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6 [term 1 FOLLOWER]: Refusing update from remote peer 22adf1fa898e47848eee442f6bf771c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:10:55.276638 20188 raft_consensus.cc:1261] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87 [term 1 FOLLOWER]: Refusing update from remote peer 22adf1fa898e47848eee442f6bf771c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:10:55.277859 20530 consensus_queue.cc:1035] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:10:55.278286 20527 consensus_queue.cc:1035] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:10:55.289806 20538 mvcc.cc:205] Tried to move back new op lower bound from 7332842108000645120 to 7332842107768819712. Current Snapshot: MvccSnapshot[applied={T|T < 7332842108000645120}]
I20260924 10:10:55.291078 20542 mvcc.cc:205] Tried to move back new op lower bound from 7332842108000645120 to 7332842107768819712. Current Snapshot: MvccSnapshot[applied={T|T < 7332842108000645120}]
I20260924 10:10:55.304167 20043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:10:55.304440 20043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:10:55.308332 20043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Sending DeleteTablet for 3 replicas of tablet 83da9e288c174ab39c7514fe485db167
I20260924 10:10:55.309585 20168 tablet_service.cc:1514] Processing DeleteTablet for tablet 83da9e288c174ab39c7514fe485db167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:10:55 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:47438
I20260924 10:10:55.309590 20436 tablet_service.cc:1514] Processing DeleteTablet for tablet 83da9e288c174ab39c7514fe485db167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:10:55 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:35474
I20260924 10:10:55.309676 20302 tablet_service.cc:1514] Processing DeleteTablet for tablet 83da9e288c174ab39c7514fe485db167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:10:55 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:47264
I20260924 10:10:55.311151 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 20105
I20260924 10:10:55.322105 20551 tablet_replica.cc:330] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4: stopping tablet replica
I20260924 10:10:55.322361 20550 tablet_replica.cc:330] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87: stopping tablet replica
I20260924 10:10:55.322659 20551 raft_consensus.cc:2229] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:10:55.322898 20550 raft_consensus.cc:2229] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:10:55.323513 20550 raft_consensus.cc:2258] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:10:55.323407 20551 raft_consensus.cc:2258] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:10:55.325532 20550 ts_tablet_manager.cc:1905] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:10:55.325539 20551 ts_tablet_manager.cc:1905] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:10:55.333398 20550 ts_tablet_manager.cc:1918] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:10:55.333679 20550 log.cc:1196] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/83da9e288c174ab39c7514fe485db167
I20260924 10:10:55.334259 20551 ts_tablet_manager.cc:1918] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:10:55.334446 20551 log.cc:1196] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/83da9e288c174ab39c7514fe485db167
I20260924 10:10:55.334488 20550 ts_tablet_manager.cc:1939] T 83da9e288c174ab39c7514fe485db167 P df0b97be3cae49b8985fd641f6375f87: Deleting consensus metadata
I20260924 10:10:55.335018 20551 ts_tablet_manager.cc:1939] T 83da9e288c174ab39c7514fe485db167 P 22adf1fa898e47848eee442f6bf771c4: Deleting consensus metadata
I20260924 10:10:55.336782 20029 catalog_manager.cc:4841] TS 22adf1fa898e47848eee442f6bf771c4 (127.6.2.195:43271): tablet 83da9e288c174ab39c7514fe485db167 (table test-table [id=2d90a818f51c4f0d90f1e241fed04914]) successfully deleted
I20260924 10:10:55.336896 20030 catalog_manager.cc:4841] TS df0b97be3cae49b8985fd641f6375f87 (127.6.2.193:37927): tablet 83da9e288c174ab39c7514fe485db167 (table test-table [id=2d90a818f51c4f0d90f1e241fed04914]) successfully deleted
I20260924 10:10:55.337900 20552 tablet_replica.cc:330] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6: stopping tablet replica
I20260924 10:10:55.338335 20552 raft_consensus.cc:2229] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:10:55.338872 20552 raft_consensus.cc:2258] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:10:55.340847 20552 ts_tablet_manager.cc:1905] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 10:10:55 dist-test-slave-s3n3 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:10:55.347450 20552 ts_tablet_manager.cc:1918] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:10:55.347620 20552 log.cc:1196] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/83da9e288c174ab39c7514fe485db167
I20260924 10:10:55.348155 20552 ts_tablet_manager.cc:1939] T 83da9e288c174ab39c7514fe485db167 P 5689d5ac27e843f29a43c321eccaced6: Deleting consensus metadata
I20260924 10:10:55.350188 20028 catalog_manager.cc:4841] TS 5689d5ac27e843f29a43c321eccaced6 (127.6.2.194:38077): tablet 83da9e288c174ab39c7514fe485db167 (table test-table [id=2d90a818f51c4f0d90f1e241fed04914]) successfully deleted
I20260924 10:10:55.353978 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:55.312443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.6.2.193:37927)
0924 10:10:55.312618 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:55.312627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:55.334884 (+ 22257us) server_negotiation.cc:407] Connection header received
0924 10:10:55.335214 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:55.335221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:55.335308 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:55.335404 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:55.337538 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.338693 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.340684 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.341334 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.341468 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:55.346966 (+ 5498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:55.346984 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:55.347004 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:55.347086 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:55.350873 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:55.351813 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:55.351820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:55.351822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:55.351960 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:55.352406 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:55.352410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:55.352412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:55.352804 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:55.353045 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:55.353466 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 10:10:55.353641 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:10:55.354645 20228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:55.472746 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 20105
I20260924 10:10:55.491288 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 20239
Sep 24 10:10:55 dist-test-slave-s3n3 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:10:55.512444 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:55.492242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.6.2.194:38077)
0924 10:10:55.492466 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:55.492472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:55.492496 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:10:55.492573 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:55.492579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:55.492682 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:55.492801 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:55.494145 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.495002 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.496529 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.496954 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.497089 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:55.506162 (+ 9073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:55.506179 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:55.506194 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:55.506236 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:55.509340 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:55.510107 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:55.510115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:55.510118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:55.510251 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:55.510637 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:55.510643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:55.510646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:55.511185 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:55.511563 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:55.511839 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 10:10:55.512063 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 10:10:55.513280 20362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:55.610337 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 20239
I20260924 10:10:55.634699 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 20373
Sep 24 10:10:55 dist-test-slave-s3n3 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:10:55.654071 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:55.635818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.6.2.195:43271)
0924 10:10:55.636025 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:55.636031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:55.636058 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:10:55.636129 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:55.636134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:55.636264 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:55.636387 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:55.637761 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.638972 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.640521 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:55.641255 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:55.641397 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:55.645943 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:55.645962 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:55.645992 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:55.646044 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:55.650417 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:55.651261 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:55.651270 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:55.651273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:55.651437 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:55.651818 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:55.651823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:55.651826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:55.652310 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:55.652652 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:55.653345 (+ 693us) server_negotiation.cc:299] Negotiation successful
0924 10:10:55.653654 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:10:55.654999 20496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:55.782701 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 20373
I20260924 10:10:55.799692 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 20012
I20260924 10:10:55.800915 20074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:55.915365 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 20012
I20260924 10:10:55.937521 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36627
--webserver_interface=127.6.2.254
--webserver_port=34357
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:36627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:10:56.277437 20558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:56.277814 20558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:56.277969 20558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:56.285410 20558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:10:56.285593 20558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:56.285707 20558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:56.285822 20558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:10:56.285903 20558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:10:56.296414 20558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:36627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=34357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:56.297987 20558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:56.299412 20558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:56.322754 20564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:56.322629 20567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:56.322695 20565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:56.323994 20558 server_base.cc:1034] running on GCE node
I20260924 10:10:56.324914 20558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:56.326577 20558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:56.327826 20558 hybrid_clock.cc:648] HybridClock initialized: now 1790244656327775 us; error 45 us; skew 500 ppm
I20260924 10:10:56.328267 20558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:56 dist-test-slave-s3n3 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:56.331936 20558 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:10:56.333860 20558 webserver.cc:460] Webserver started at http://127.6.2.254:34357/ using document root <none> and password file <none>
I20260924 10:10:56.334631 20558 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:56.334816 20558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:56.340903 20558 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:10:56.344184 20574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:56.345714 20558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:10:56.345948 20558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "ddf119c5899e4b3080bdf81412d3f7d9"
format_stamp: "Formatted at 2026-09-24 10:10:52 on dist-test-slave-s3n3"
server_key: "a5f735ed2db9197c6df3c754805fd63a"
server_key_iv: "2cca7f5224c9f56c240dd4a51efd056d"
server_key_version: "encryptionkey@0"
I20260924 10:10:56.347071 20558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:56.379674 20558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:56.384575 20558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:56.384904 20558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:56.405009 20558 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:36627
I20260924 10:10:56.405027 20625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:36627 every 8 connection(s)
I20260924 10:10:56.406594 20558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:10:56.411985 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 20558
I20260924 10:10:56.412834 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:37927
--local_ip_for_outbound_sockets=127.6.2.193
--tserver_master_addrs=127.6.2.254:36627
--webserver_port=36011
--webserver_interface=127.6.2.193
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:10:56.414075 20626 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:10:56.417510 20626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Bootstrap starting.
I20260924 10:10:56.437598 20626 log.cc:824] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:56.450006 20626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:10:56.450651 20626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Bootstrap complete.
I20260924 10:10:56.460330 20626 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } }
I20260924 10:10:56.461323 20626 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf119c5899e4b3080bdf81412d3f7d9, State: Initialized, Role: FOLLOWER
I20260924 10:10:56.462015 20626 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } }
I20260924 10:10:56.462275 20626 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:10:56.462415 20626 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:10:56.462594 20626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:10:56.466533 20626 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } }
I20260924 10:10:56.467049 20626 leader_election.cc:302] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf119c5899e4b3080bdf81412d3f7d9; no voters:
I20260924 10:10:56.467775 20626 leader_election.cc:288] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:10:56.467916 20630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:10:56.469098 20630 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 2 LEADER]: Becoming Leader. State: Replica: ddf119c5899e4b3080bdf81412d3f7d9, State: Running, Role: LEADER
I20260924 10:10:56.469712 20630 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } }
I20260924 10:10:56.470278 20626 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:10:56.472406 20632 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf119c5899e4b3080bdf81412d3f7d9. Latest consensus state: current_term: 2 leader_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } } }
I20260924 10:10:56.472807 20632 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:56.473330 20631 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } } }
I20260924 10:10:56.473604 20631 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:56.483011 20636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:10:56.491696 20636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:10:56.493227 20636 catalog_manager.cc:1259] Loaded cluster ID: 66b602989e99440ca5b1e95c573f3fe7
I20260924 10:10:56.493396 20636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:10:56.496656 20636 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 10:10:56.515586 20636 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 10:10:56.541816 20636 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:10:56.544857 20636 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Loaded TSK: 0
I20260924 10:10:56.546962 20636 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:10:56.793035 20628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:56.793375 20628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:56.793543 20628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:56.801167 20628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:56.801321 20628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:56.801509 20628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:10:56.811997 20628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:37927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=36011
--tserver_master_addrs=127.6.2.254:36627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:56.813372 20628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:56.814687 20628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:56.829849 20653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:56.829912 20654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:56.829949 20656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:56.830741 20628 server_base.cc:1034] running on GCE node
I20260924 10:10:56.831523 20628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:56.832577 20628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:56.833858 20628 hybrid_clock.cc:648] HybridClock initialized: now 1790244656833777 us; error 72 us; skew 500 ppm
I20260924 10:10:56.834204 20628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:56 dist-test-slave-s3n3 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:56.837453 20628 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:10:56.839195 20628 webserver.cc:460] Webserver started at http://127.6.2.193:36011/ using document root <none> and password file <none>
I20260924 10:10:56.839905 20628 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:56.840044 20628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:56.845659 20628 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:10:56.848526 20663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:56.849924 20628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:10:56.850086 20628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "df0b97be3cae49b8985fd641f6375f87"
format_stamp: "Formatted at 2026-09-24 10:10:53 on dist-test-slave-s3n3"
server_key: "28fee85218e345f36a31c71d829ea65b"
server_key_iv: "584ca27498ffec8b40a8c72e1fb3419c"
server_key_version: "encryptionkey@0"
I20260924 10:10:56.851181 20628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:56.870832 20628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:56.876256 20628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:56.876549 20628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:56.877777 20628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:10:56.879479 20628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:10:56.879588 20628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:56.879698 20628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:10:56.879770 20628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:56.913203 20628 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:37927
I20260924 10:10:56.913311 20775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:37927 every 8 connection(s)
I20260924 10:10:56.914640 20628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:10:56.918115 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 20628
I20260924 10:10:56.918965 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:38077
--local_ip_for_outbound_sockets=127.6.2.194
--tserver_master_addrs=127.6.2.254:36627
--webserver_port=45043
--webserver_interface=127.6.2.194
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:10:56 dist-test-slave-s3n3 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:56.942364 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:56.918049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:55145)
0924 10:10:56.919131 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:56.919183 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:56.920916 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:56.922401 (+ 1485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:56.922416 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:56.923374 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:56.924188 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:56.924229 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:56.926108 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:56.926113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:56.927568 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:56.927582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:56.929207 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:56.929212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:56.929401 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:56.930128 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:56.930161 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:56.932629 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:56.937864 (+ 5235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:56.937872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:56.937884 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:56.938348 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:56.938902 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:56.938905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:56.938907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:56.939162 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:56.940240 (+ 1078us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:56.940246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:56.940617 (+ 371us) client_negotiation.cc:769] Sending connection context
0924 10:10:56.940863 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 10:10:56.941273 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":560,"threads_started":1}
I20260924 10:10:56.942906 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:56.917960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:55145 (local address 127.6.2.254:36627)
0924 10:10:56.918723 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:56.918739 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:56.921204 (+ 2465us) server_negotiation.cc:407] Connection header received
0924 10:10:56.921500 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:56.921511 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:56.921839 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:56.922112 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:56.924482 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:56.925918 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:56.927733 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:56.928932 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:56.929133 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:56.933055 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:56.933075 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:56.933100 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:56.933154 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:56.937694 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:56.938512 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:56.938525 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:56.938535 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:56.938773 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:56.939292 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:56.939300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:56.939306 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:56.940079 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:56.940327 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:56.941164 (+ 837us) server_negotiation.cc:299] Negotiation successful
0924 10:10:56.941657 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":331,"threads_started":1}
I20260924 10:10:56.945735 20776 heartbeater.cc:347] Connected to a master server at 127.6.2.254:36627
I20260924 10:10:56.946043 20776 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:56.946810 20776 heartbeater.cc:511] Master 127.6.2.254:36627 requested a full tablet report, sending...
I20260924 10:10:56.949059 20591 ts_manager.cc:194] Registered new tserver with Master: df0b97be3cae49b8985fd641f6375f87 (127.6.2.193:37927)
I20260924 10:10:56.950506 20591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:55145
W20260924 10:10:57.250401 20780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:57.250751 20780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:57.250908 20780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:57.258555 20780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:57.258728 20780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:57.258898 20780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:10:57.269047 20780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:38077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=45043
--tserver_master_addrs=127.6.2.254:36627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:57.270453 20780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:57.271837 20780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:57.284780 20785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:57.284924 20786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:57.284902 20788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:57.286217 20780 server_base.cc:1034] running on GCE node
I20260924 10:10:57.286835 20780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:57.287871 20780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:57.289111 20780 hybrid_clock.cc:648] HybridClock initialized: now 1790244657289050 us; error 56 us; skew 500 ppm
I20260924 10:10:57.289482 20780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:57 dist-test-slave-s3n3 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244657, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:57.292809 20780 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:10:57.294441 20780 webserver.cc:460] Webserver started at http://127.6.2.194:45043/ using document root <none> and password file <none>
I20260924 10:10:57.295292 20780 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:57.295459 20780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:57.301781 20780 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 10:10:57.304776 20795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:57.306341 20780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:10:57.306528 20780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "5689d5ac27e843f29a43c321eccaced6"
format_stamp: "Formatted at 2026-09-24 10:10:53 on dist-test-slave-s3n3"
server_key: "c205be34fe412fbcb2783337b9b2ca07"
server_key_iv: "b2bd0a33f04dd7c9c53eff6b3c41a7d5"
server_key_version: "encryptionkey@0"
I20260924 10:10:57.307605 20780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:57.336200 20780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:57.340559 20780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:57.340859 20780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:57.341960 20780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:10:57.343504 20780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:10:57.343633 20780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:57.343767 20780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:10:57.343864 20780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:57.377941 20780 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:38077
I20260924 10:10:57.378028 20907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:38077 every 8 connection(s)
I20260924 10:10:57.379441 20780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:10:57.386077 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 20780
I20260924 10:10:57.386711 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:43271
--local_ip_for_outbound_sockets=127.6.2.195
--tserver_master_addrs=127.6.2.254:36627
--webserver_port=36277
--webserver_interface=127.6.2.195
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:10:57 dist-test-slave-s3n3 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244657, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:57.400449 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:57.381974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:53303 (local address 127.6.2.254:36627)
0924 10:10:57.382210 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:57.382217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:57.383413 (+ 1196us) server_negotiation.cc:407] Connection header received
0924 10:10:57.385196 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:57.385201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:57.385269 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:57.385359 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:57.387008 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.388199 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.390000 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.390796 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.390948 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:57.394548 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:57.394553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:57.394556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:57.394590 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:57.397072 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:57.398017 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:57.398021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:57.398023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:57.398176 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:57.398647 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:57.398650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:57.398652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:57.398920 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:57.399032 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:57.400013 (+ 981us) server_negotiation.cc:299] Negotiation successful
0924 10:10:57.400204 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 10:10:57.402015 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:57.382439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:53303)
0924 10:10:57.383246 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:57.383293 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:57.384946 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:57.385559 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:57.385567 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:57.386147 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:57.386767 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:57.386785 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.388386 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.388390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:57.389849 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:57.389862 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.391014 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.391019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:57.391288 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:57.391996 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:57.392019 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:57.394316 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:57.397218 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:57.397231 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:57.397250 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:57.397892 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:57.398285 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:57.398293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:57.398299 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:57.398538 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:57.399050 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:57.399089 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:57.399574 (+ 485us) client_negotiation.cc:769] Sending connection context
0924 10:10:57.399920 (+ 346us) client_negotiation.cc:240] Negotiation successful
0924 10:10:57.400515 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":309,"threads_started":1}
I20260924 10:10:57.403656 20908 heartbeater.cc:347] Connected to a master server at 127.6.2.254:36627
I20260924 10:10:57.403998 20908 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:57.404980 20908 heartbeater.cc:511] Master 127.6.2.254:36627 requested a full tablet report, sending...
I20260924 10:10:57.406870 20591 ts_manager.cc:194] Registered new tserver with Master: 5689d5ac27e843f29a43c321eccaced6 (127.6.2.194:38077)
I20260924 10:10:57.408016 20591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:53303
W20260924 10:10:57.716867 20911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:57.717200 20911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:57.717372 20911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:57.726282 20911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:57.726430 20911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:57.726611 20911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:10:57.736842 20911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:43271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=36277
--tserver_master_addrs=127.6.2.254:36627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:57.738197 20911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:57.739550 20911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:57.753118 20919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:57.753098 20916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:57.753181 20917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:57.753870 20911 server_base.cc:1034] running on GCE node
I20260924 10:10:57.754566 20911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:57.755628 20911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:57.756855 20911 hybrid_clock.cc:648] HybridClock initialized: now 1790244657756811 us; error 36 us; skew 500 ppm
I20260924 10:10:57.757226 20911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:57 dist-test-slave-s3n3 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244657, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:57.760703 20911 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:10:57.762320 20911 webserver.cc:460] Webserver started at http://127.6.2.195:36277/ using document root <none> and password file <none>
I20260924 10:10:57.763026 20911 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:57.763240 20911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:57.768759 20911 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:10:57.771865 20926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:57.773222 20911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:10:57.773407 20911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "22adf1fa898e47848eee442f6bf771c4"
format_stamp: "Formatted at 2026-09-24 10:10:54 on dist-test-slave-s3n3"
server_key: "940a40226933b0f01ce0aeeeeda7e4f1"
server_key_iv: "82ea5cacf9b5c713e2fdc3f7c1af7a93"
server_key_version: "encryptionkey@0"
I20260924 10:10:57.774387 20911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:57.791463 20911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:57.795928 20911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:57.796216 20911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:57.797230 20911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:10:57.798662 20911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:10:57.798748 20911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:57.798893 20911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:10:57.798965 20911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:57.832163 20911 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:43271
I20260924 10:10:57.832197 21038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:43271 every 8 connection(s)
I20260924 10:10:57.833648 20911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:10:57.842574 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 20911
Sep 24 10:10:57 dist-test-slave-s3n3 krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244657, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:10:57.856586 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:57.836415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:33377 (local address 127.6.2.254:36627)
0924 10:10:57.836657 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:57.836664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:57.837917 (+ 1253us) server_negotiation.cc:407] Connection header received
0924 10:10:57.839799 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:57.839803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:57.839872 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:57.839966 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:57.841535 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.842369 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.844531 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.845153 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.845250 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:57.849405 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:57.849412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:57.849417 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:57.849457 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:57.852741 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:57.853541 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:57.853548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:57.853553 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:57.853715 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:57.854236 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:57.854243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:57.854249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:57.854649 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:57.854832 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:57.855914 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0924 10:10:57.856204 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 10:10:57.857295 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:57.836934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:33377)
0924 10:10:57.837740 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:57.837777 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:57.839531 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:57.840197 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:57.840213 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:57.840736 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:10:57.841354 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:57.841371 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.842529 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.842535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:57.844386 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:57.844402 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.845405 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.845415 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:57.845768 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:57.846570 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:10:57.846604 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:10:57.849166 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:10:57.852883 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:57.852893 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:57.852911 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:57.853401 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:57.853834 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:10:57.853839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:10:57.853844 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:10:57.854065 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:10:57.854861 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:10:57.854872 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:10:57.855500 (+ 628us) client_negotiation.cc:769] Sending connection context
0924 10:10:57.855758 (+ 258us) client_negotiation.cc:240] Negotiation successful
0924 10:10:57.856160 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":331,"threads_started":1}
I20260924 10:10:57.858939 21039 heartbeater.cc:347] Connected to a master server at 127.6.2.254:36627
I20260924 10:10:57.859313 21039 heartbeater.cc:464] Registering TS with master...
I20260924 10:10:57.860204 21039 heartbeater.cc:511] Master 127.6.2.254:36627 requested a full tablet report, sending...
I20260924 10:10:57.860282 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:57.843791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55538 (local address 127.6.2.254:36627)
0924 10:10:57.844535 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:57.844540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:57.844554 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:10:57.844601 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:57.844606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:57.844684 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:57.844791 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:57.846664 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.847565 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.849260 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.850023 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.850167 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:57.853095 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:57.853100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:57.853103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:57.853130 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:57.856670 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:57.858213 (+ 1543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:57.858223 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:57.858225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:57.858346 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:57.858721 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:57.858727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:57.858731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:57.859111 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:57.859396 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:57.859677 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 10:10:57.859900 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":479,"threads_started":1}
I20260924 10:10:57.861965 20591 ts_manager.cc:194] Registered new tserver with Master: 22adf1fa898e47848eee442f6bf771c4 (127.6.2.195:43271)
I20260924 10:10:57.863143 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:10:57.863150 20591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:33377
I20260924 10:10:57.870042 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:57.864186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.6.2.254:36627)
0924 10:10:57.864364 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:57.864371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:57.864899 (+ 528us) server_negotiation.cc:407] Connection header received
0924 10:10:57.865103 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:57.865107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:57.865231 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:57.865350 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:10:57.865826 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.866478 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.867992 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.868405 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.868498 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:57.868942 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:10:57.869385 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:10:57.869494 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:57.869643 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 10:10:57.869707 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:10:57.870894 20591 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:10:57.878449 20591 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:10:57.905494 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:57.891365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.6.2.195:43271)
0924 10:10:57.893069 (+ 1704us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:57.893082 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:57.893683 (+ 601us) server_negotiation.cc:407] Connection header received
0924 10:10:57.893992 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:57.894000 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:57.894324 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:57.894527 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:57.895832 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.898021 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.901581 (+ 3560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.903383 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.903504 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:57.903822 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:57.904718 (+ 896us) server_negotiation.cc:299] Negotiation successful
0924 10:10:57.904962 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1518,"thread_start_us":1370,"threads_started":1}
I20260924 10:10:57.906493 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:57.890991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:43271 (local address 127.0.0.1:39158)
0924 10:10:57.893577 (+ 2586us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:57.893595 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:57.893792 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:57.894938 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:57.894949 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:57.894990 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:57.895573 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:57.895589 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.898311 (+ 2722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.898319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:57.901339 (+ 3020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:57.901363 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.904056 (+ 2693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.904065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:57.904369 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:57.904449 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 10:10:57.905831 (+ 1382us) client_negotiation.cc:240] Negotiation successful
0924 10:10:57.906063 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2456,"thread_start_us":348,"threads_started":1}
I20260924 10:10:57.910817 20974 tablet_service.cc:1467] Processing CreateTablet for tablet 49f98624062f4c368fdd838f0fd8b06f (DEFAULT_TABLE table=test-table [id=663b6b2c94614e8c9359a0f07d59a73f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:10:57.912438 20974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f98624062f4c368fdd838f0fd8b06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:57.917507 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:57.892972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:37927 (local address 127.0.0.1:55954)
0924 10:10:57.908659 (+ 15687us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:57.908678 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:57.908835 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:57.909429 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:57.909433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:57.909485 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:57.909854 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:57.909866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.911819 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.911825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:57.914226 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:57.914254 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.916567 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.916578 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:57.916790 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:57.916824 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:10:57.916915 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 10:10:57.917057 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15560,"mutex_wait_us":98,"spinlock_wait_cycles":1408,"thread_start_us":15409,"threads_started":1}
I20260924 10:10:57.918244 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:57.893915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55954 (local address 127.6.2.193:37927)
0924 10:10:57.908215 (+ 14300us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:57.908224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:57.908807 (+ 583us) server_negotiation.cc:407] Connection header received
0924 10:10:57.908990 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:57.909001 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:57.909217 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:57.909423 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:57.910068 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.911642 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.914443 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.916066 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.916188 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:57.916402 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:57.917743 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0924 10:10:57.917908 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14146,"thread_start_us":13813,"threads_started":1}
I20260924 10:10:57.918303 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:57.890779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:38077 (local address 127.0.0.1:39386)
0924 10:10:57.891906 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:57.891955 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:57.892163 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:57.909203 (+ 17040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:57.909207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:57.909247 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:57.909756 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:57.909768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.912400 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.912405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:57.914537 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:57.914554 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.917410 (+ 2856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.917416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:57.917595 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:57.917630 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:10:57.917717 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 10:10:57.917927 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":8,"thread_start_us":2527,"threads_started":1}
I20260924 10:10:57.919442 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:57.891154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.6.2.194:38077)
0924 10:10:57.908529 (+ 17375us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:57.908538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:57.908576 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:10:57.908694 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:57.908700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:57.908971 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:57.909930 (+ 959us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:57.910242 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.912203 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.914735 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:57.917162 (+ 2427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:57.918435 (+ 1273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:57.918897 (+ 462us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:57.918964 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 10:10:57.919152 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17187,"thread_start_us":17886,"threads_started":1}
I20260924 10:10:57.922433 20711 tablet_service.cc:1467] Processing CreateTablet for tablet 49f98624062f4c368fdd838f0fd8b06f (DEFAULT_TABLE table=test-table [id=663b6b2c94614e8c9359a0f07d59a73f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:10:57.923858 20711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f98624062f4c368fdd838f0fd8b06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:57.924078 20843 tablet_service.cc:1467] Processing CreateTablet for tablet 49f98624062f4c368fdd838f0fd8b06f (DEFAULT_TABLE table=test-table [id=663b6b2c94614e8c9359a0f07d59a73f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:10:57.925069 20843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f98624062f4c368fdd838f0fd8b06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:10:57.928268 21055 tablet_bootstrap.cc:492] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4: Bootstrap starting.
I20260924 10:10:57.932333 21055 tablet_bootstrap.cc:654] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:57.933604 21055 log.cc:824] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:57.936846 21055 tablet_bootstrap.cc:492] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4: No bootstrap required, opened a new log
I20260924 10:10:57.937130 21055 ts_tablet_manager.cc:1397] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 10:10:57.937544 21057 tablet_bootstrap.cc:492] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87: Bootstrap starting.
I20260924 10:10:57.940671 21057 tablet_bootstrap.cc:654] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:57.941725 21057 log.cc:824] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:57.944500 21057 tablet_bootstrap.cc:492] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87: No bootstrap required, opened a new log
I20260924 10:10:57.944751 21057 ts_tablet_manager.cc:1397] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260924 10:10:57.947222 21058 tablet_bootstrap.cc:492] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6: Bootstrap starting.
I20260924 10:10:57.948077 21055 raft_consensus.cc:348] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } } peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } }
I20260924 10:10:57.948438 21055 raft_consensus.cc:374] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:57.948534 21055 raft_consensus.cc:729] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22adf1fa898e47848eee442f6bf771c4, State: Initialized, Role: FOLLOWER
I20260924 10:10:57.949275 21055 consensus_queue.cc:260] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } } peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } }
I20260924 10:10:57.950826 21058 tablet_bootstrap.cc:654] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6: Neither blocks nor log segments found. Creating new log.
I20260924 10:10:57.952095 21058 log.cc:824] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:57.953481 20776 heartbeater.cc:503] Master 127.6.2.254:36627 was elected leader, sending a full tablet report...
I20260924 10:10:57.954154 21055 ts_tablet_manager.cc:1428] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4: Time spent starting tablet: real 0.017s user 0.008s sys 0.009s
I20260924 10:10:57.954221 21057 raft_consensus.cc:348] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } } peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } }
I20260924 10:10:57.954711 21057 raft_consensus.cc:374] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:57.954828 21057 raft_consensus.cc:729] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0b97be3cae49b8985fd641f6375f87, State: Initialized, Role: FOLLOWER
I20260924 10:10:57.955030 21039 heartbeater.cc:503] Master 127.6.2.254:36627 was elected leader, sending a full tablet report...
I20260924 10:10:57.955333 21058 tablet_bootstrap.cc:492] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6: No bootstrap required, opened a new log
I20260924 10:10:57.955607 21058 ts_tablet_manager.cc:1397] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260924 10:10:57.955459 21057 consensus_queue.cc:260] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } } peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } }
I20260924 10:10:57.958624 21057 ts_tablet_manager.cc:1428] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260924 10:10:57.964305 21058 raft_consensus.cc:348] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } } peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } }
I20260924 10:10:57.964680 21058 raft_consensus.cc:374] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:10:57.964807 21058 raft_consensus.cc:729] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5689d5ac27e843f29a43c321eccaced6, State: Initialized, Role: FOLLOWER
I20260924 10:10:57.965421 21058 consensus_queue.cc:260] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } } peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } }
I20260924 10:10:57.966419 20908 heartbeater.cc:503] Master 127.6.2.254:36627 was elected leader, sending a full tablet report...
I20260924 10:10:57.966890 21058 ts_tablet_manager.cc:1428] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260924 10:10:58.085299 21040 tablet.cc:2369] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:10:58.131460 20909 tablet.cc:2369] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:10:58.167435 20777 tablet.cc:2369] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:10:58.365494 21062 raft_consensus.cc:482] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:10:58.365813 21062 raft_consensus.cc:504] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } } peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } }
I20260924 10:10:58.367826 21062 leader_election.cc:288] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5689d5ac27e843f29a43c321eccaced6 (127.6.2.194:38077), 22adf1fa898e47848eee442f6bf771c4 (127.6.2.195:43271)
I20260924 10:10:58.377017 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:58.368420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:42371 (local address 127.6.2.195:43271)
0924 10:10:58.368593 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:58.368597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:58.369299 (+ 702us) server_negotiation.cc:407] Connection header received
0924 10:10:58.369504 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:58.369508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:58.369568 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:58.369666 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:58.370412 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.371269 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.374420 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.375747 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.375885 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:58.375991 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:58.376595 (+ 604us) server_negotiation.cc:299] Negotiation successful
0924 10:10:58.376788 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:10:58.376996 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:58.368203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:53055 (local address 127.6.2.194:38077)
0924 10:10:58.368445 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:58.368454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:58.369245 (+ 791us) server_negotiation.cc:407] Connection header received
0924 10:10:58.369478 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:58.369482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:58.369542 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:58.369638 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:10:58.370527 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.371425 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.374418 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.375731 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.375885 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:58.375988 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:58.376594 (+ 606us) server_negotiation.cc:299] Negotiation successful
0924 10:10:58.376736 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:10:58.377231 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:58.368353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:43271 (local address 127.6.2.193:42371)
0924 10:10:58.369192 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:58.369207 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:58.369356 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:58.369680 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:58.369684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:58.369709 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:58.370217 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:58.370228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.371514 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.371520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:58.374224 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:58.374240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.376031 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.376036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:58.376331 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:58.376438 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 10:10:58.376537 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 10:10:58.376757 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":341,"thread_start_us":547,"threads_started":1}
I20260924 10:10:58.377254 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:58.368227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:38077 (local address 127.6.2.193:53055)
0924 10:10:58.369133 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:58.369150 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:10:58.369327 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:10:58.369663 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:10:58.369667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:10:58.369692 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:10:58.370313 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:58.370326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.371529 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.371534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:58.374224 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:10:58.374240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.376017 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.376024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:10:58.376335 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:58.376442 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 10:10:58.376537 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:10:58.376757 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":287,"thread_start_us":432,"threads_started":1}
I20260924 10:10:58.378736 20994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f98624062f4c368fdd838f0fd8b06f" candidate_uuid: "df0b97be3cae49b8985fd641f6375f87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22adf1fa898e47848eee442f6bf771c4" is_pre_election: true
I20260924 10:10:58.378962 20863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f98624062f4c368fdd838f0fd8b06f" candidate_uuid: "df0b97be3cae49b8985fd641f6375f87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5689d5ac27e843f29a43c321eccaced6" is_pre_election: true
I20260924 10:10:58.379252 20994 raft_consensus.cc:2454] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0b97be3cae49b8985fd641f6375f87 in term 0.
I20260924 10:10:58.379539 20863 raft_consensus.cc:2454] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0b97be3cae49b8985fd641f6375f87 in term 0.
I20260924 10:10:58.379868 20665 leader_election.cc:302] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22adf1fa898e47848eee442f6bf771c4, df0b97be3cae49b8985fd641f6375f87; no voters:
I20260924 10:10:58.380322 21062 raft_consensus.cc:2781] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:10:58.380491 21062 raft_consensus.cc:482] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:10:58.380612 21062 raft_consensus.cc:3037] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:58.383817 21062 raft_consensus.cc:504] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } } peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } }
I20260924 10:10:58.384497 21062 leader_election.cc:288] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [CANDIDATE]: Term 1 election: Requested vote from peers 5689d5ac27e843f29a43c321eccaced6 (127.6.2.194:38077), 22adf1fa898e47848eee442f6bf771c4 (127.6.2.195:43271)
I20260924 10:10:58.385035 20863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f98624062f4c368fdd838f0fd8b06f" candidate_uuid: "df0b97be3cae49b8985fd641f6375f87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5689d5ac27e843f29a43c321eccaced6"
I20260924 10:10:58.385241 20863 raft_consensus.cc:3037] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:58.387315 20994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f98624062f4c368fdd838f0fd8b06f" candidate_uuid: "df0b97be3cae49b8985fd641f6375f87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22adf1fa898e47848eee442f6bf771c4"
I20260924 10:10:58.387562 20994 raft_consensus.cc:3037] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:10:58.388479 20863 raft_consensus.cc:2454] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0b97be3cae49b8985fd641f6375f87 in term 1.
I20260924 10:10:58.388929 20664 leader_election.cc:302] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5689d5ac27e843f29a43c321eccaced6, df0b97be3cae49b8985fd641f6375f87; no voters:
I20260924 10:10:58.389233 21062 raft_consensus.cc:2781] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:10:58.390161 20994 raft_consensus.cc:2454] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0b97be3cae49b8985fd641f6375f87 in term 1.
I20260924 10:10:58.390640 21062 raft_consensus.cc:686] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [term 1 LEADER]: Becoming Leader. State: Replica: df0b97be3cae49b8985fd641f6375f87, State: Running, Role: LEADER
I20260924 10:10:58.391500 21062 consensus_queue.cc:237] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } } peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } }
I20260924 10:10:58.396646 20591 catalog_manager.cc:5495] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 reported cstate change: term changed from 0 to 1, leader changed from <none> to df0b97be3cae49b8985fd641f6375f87 (127.6.2.193). New cstate: current_term: 1 leader_uuid: "df0b97be3cae49b8985fd641f6375f87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df0b97be3cae49b8985fd641f6375f87" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 37927 } health_report { overall_health: HEALTHY } } }
I20260924 10:10:58.476728 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:58.470827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.6.2.193:37927)
0924 10:10:58.471318 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:58.471323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:58.471378 (+ 55us) server_negotiation.cc:407] Connection header received
0924 10:10:58.471611 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:58.471615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:58.471804 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:58.471953 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:10:58.472296 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.473069 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.474451 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.474921 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.475026 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:58.475593 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:10:58.476173 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:10:58.476289 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:58.476403 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 10:10:58.476469 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":272,"threads_started":1}
I20260924 10:10:58.485004 20863 raft_consensus.cc:1261] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6 [term 1 FOLLOWER]: Refusing update from remote peer df0b97be3cae49b8985fd641f6375f87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:10:58.485030 20994 raft_consensus.cc:1261] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4 [term 1 FOLLOWER]: Refusing update from remote peer df0b97be3cae49b8985fd641f6375f87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:10:58.485992 21062 consensus_queue.cc:1035] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22adf1fa898e47848eee442f6bf771c4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:10:58.486330 21066 consensus_queue.cc:1035] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5689d5ac27e843f29a43c321eccaced6" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:10:58.498732 21077 mvcc.cc:205] Tried to move back new op lower bound from 7332842121144840192 to 7332842120776175616. Current Snapshot: MvccSnapshot[applied={T|T < 7332842121144840192}]
I20260924 10:10:58.511330 20590 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:10:58.511574 20590 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:10:58.514963 20590 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Sending DeleteTablet for 3 replicas of tablet 49f98624062f4c368fdd838f0fd8b06f
I20260924 10:10:58.516175 20974 tablet_service.cc:1514] Processing DeleteTablet for tablet 49f98624062f4c368fdd838f0fd8b06f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:10:58 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:39158
I20260924 10:10:58.516175 20843 tablet_service.cc:1514] Processing DeleteTablet for tablet 49f98624062f4c368fdd838f0fd8b06f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:10:58 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:39386
I20260924 10:10:58.517102 21087 tablet_replica.cc:330] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4: stopping tablet replica
I20260924 10:10:58.517231 20711 tablet_service.cc:1514] Processing DeleteTablet for tablet 49f98624062f4c368fdd838f0fd8b06f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:10:58 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:55954
I20260924 10:10:58.517683 21087 raft_consensus.cc:2229] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:10:58.518242 21087 raft_consensus.cc:2258] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:10:58.518261 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 20628
I20260924 10:10:58.520334 21087 ts_tablet_manager.cc:1905] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:10:58.527798 21087 ts_tablet_manager.cc:1918] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:10:58.528523 21086 tablet_replica.cc:330] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6: stopping tablet replica
I20260924 10:10:58.529052 21086 raft_consensus.cc:2229] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:10:58.529592 21086 raft_consensus.cc:2258] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:10:58.530051 21087 log.cc:1196] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/49f98624062f4c368fdd838f0fd8b06f
I20260924 10:10:58.531652 21086 ts_tablet_manager.cc:1905] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:10:58.538981 21086 ts_tablet_manager.cc:1918] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:10:58.539183 21086 log.cc:1196] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/49f98624062f4c368fdd838f0fd8b06f
I20260924 10:10:58.539769 21086 ts_tablet_manager.cc:1939] T 49f98624062f4c368fdd838f0fd8b06f P 5689d5ac27e843f29a43c321eccaced6: Deleting consensus metadata
I20260924 10:10:58.539803 21088 tablet_replica.cc:330] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87: stopping tablet replica
I20260924 10:10:58.540215 21088 raft_consensus.cc:2229] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:10:58.540812 21088 raft_consensus.cc:2258] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:10:58.541718 20575 catalog_manager.cc:4841] TS 5689d5ac27e843f29a43c321eccaced6 (127.6.2.194:38077): tablet 49f98624062f4c368fdd838f0fd8b06f (table test-table [id=663b6b2c94614e8c9359a0f07d59a73f]) successfully deleted
I20260924 10:10:58.542172 21087 ts_tablet_manager.cc:1939] T 49f98624062f4c368fdd838f0fd8b06f P 22adf1fa898e47848eee442f6bf771c4: Deleting consensus metadata
I20260924 10:10:58.543967 20576 catalog_manager.cc:4841] TS 22adf1fa898e47848eee442f6bf771c4 (127.6.2.195:43271): tablet 49f98624062f4c368fdd838f0fd8b06f (table test-table [id=663b6b2c94614e8c9359a0f07d59a73f]) successfully deleted
I20260924 10:10:58.543995 21088 ts_tablet_manager.cc:1905] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:10:58.549860 21088 ts_tablet_manager.cc:1918] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:10:58.550052 21088 log.cc:1196] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/49f98624062f4c368fdd838f0fd8b06f
I20260924 10:10:58.550650 21088 ts_tablet_manager.cc:1939] T 49f98624062f4c368fdd838f0fd8b06f P df0b97be3cae49b8985fd641f6375f87: Deleting consensus metadata
I20260924 10:10:58.552434 20577 catalog_manager.cc:4841] TS df0b97be3cae49b8985fd641f6375f87 (127.6.2.193:37927): tablet 49f98624062f4c368fdd838f0fd8b06f (table test-table [id=663b6b2c94614e8c9359a0f07d59a73f]) successfully deleted
I20260924 10:10:58.557902 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:58.519590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.6.2.193:37927)
0924 10:10:58.521921 (+ 2331us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:58.521930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:58.537223 (+ 15293us) server_negotiation.cc:407] Connection header received
0924 10:10:58.537588 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:58.537598 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:58.537734 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:58.537833 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:58.542536 (+ 4703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.544919 (+ 2383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.546908 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.547587 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.547739 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:58.551224 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:58.551236 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:58.551253 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:58.551298 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:58.554937 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:58.555873 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:58.555881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:58.555883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:58.556021 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:58.556443 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:58.556446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:58.556448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:58.556778 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:58.556995 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:58.557396 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 10:10:58.557582 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2165}
I20260924 10:10:58.558619 20771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:58.659006 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 20628
I20260924 10:10:58.675446 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 20780
I20260924 10:10:58.690163 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:58.676365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39392 (local address 127.6.2.194:38077)
0924 10:10:58.676565 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:58.676571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:58.676593 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:10:58.676664 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:58.676672 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:58.676782 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:58.676918 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:58.678210 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.678972 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.680376 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.680791 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.680955 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:58.683856 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:58.683869 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:58.683885 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:58.683918 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:58.686844 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:58.687860 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:58.687869 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:58.687872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:58.688010 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:58.688436 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:58.688443 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:58.688448 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:58.688858 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:58.689202 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:58.689527 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 10:10:58.689783 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:10:58.690923 20903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:58.783254 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 20780
I20260924 10:10:58.800712 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 20911
I20260924 10:10:58.815166 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:58.801549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.6.2.195:43271)
0924 10:10:58.801752 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 10:10:58.801759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:10:58.801807 (+ 48us) server_negotiation.cc:407] Connection header received
0924 10:10:58.802026 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:10:58.802032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:10:58.802105 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:10:58.802238 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:10:58.803660 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.804423 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.805761 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:10:58.806256 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:10:58.806390 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:10:58.809102 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:10:58.809118 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:10:58.809135 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:10:58.809168 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:10:58.812230 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:58.812942 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:58.812957 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:58.812960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:58.813094 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:10:58.813485 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:10:58.813491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:10:58.813495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:10:58.813867 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:10:58.814174 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 10:10:58.814497 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 10:10:58.814740 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 10:10:58.815958 21034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:58.904685 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 20911
I20260924 10:10:58.923317 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 20558
I20260924 10:10:58.924583 20621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:10:59.039366 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 20558
I20260924 10:10:59.061033 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36627
--webserver_interface=127.6.2.254
--webserver_port=34357
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:36627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:10:59.385507 21094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:59.385847 21094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:59.386004 21094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:59.393425 21094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:10:59.393581 21094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:59.393679 21094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:59.393776 21094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:10:59.393851 21094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:10:59.403811 21094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:36627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=34357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:59.405184 21094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:59.406543 21094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:59.418632 21099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:59.418743 21102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:59.418730 21100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:59.419715 21094 server_base.cc:1034] running on GCE node
I20260924 10:10:59.420490 21094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:59.421962 21094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:59.423183 21094 hybrid_clock.cc:648] HybridClock initialized: now 1790244659423133 us; error 57 us; skew 500 ppm
I20260924 10:10:59.423533 21094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:59 dist-test-slave-s3n3 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244659, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:59.426614 21094 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:10:59.428241 21094 webserver.cc:460] Webserver started at http://127.6.2.254:34357/ using document root <none> and password file <none>
I20260924 10:10:59.428951 21094 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:59.429101 21094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:59.434628 21094 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:10:59.439373 21109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:59.440838 21094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:10:59.441051 21094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "ddf119c5899e4b3080bdf81412d3f7d9"
format_stamp: "Formatted at 2026-09-24 10:10:52 on dist-test-slave-s3n3"
server_key: "a5f735ed2db9197c6df3c754805fd63a"
server_key_iv: "2cca7f5224c9f56c240dd4a51efd056d"
server_key_version: "encryptionkey@0"
I20260924 10:10:59.442086 21094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:59.456962 21094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:59.462059 21094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:59.462370 21094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:59.481122 21094 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:36627
I20260924 10:10:59.481159 21160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:36627 every 8 connection(s)
I20260924 10:10:59.482575 21094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:10:59.483248 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 21094
I20260924 10:10:59.484261 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:37927
--local_ip_for_outbound_sockets=127.6.2.193
--tserver_master_addrs=127.6.2.254:36627
--webserver_port=36011
--webserver_interface=127.6.2.193
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:10:59.489799 21161 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:10:59.493098 21161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Bootstrap starting.
I20260924 10:10:59.513288 21161 log.cc:824] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Log is configured to *not* fsync() on all Append() calls
I20260924 10:10:59.528981 21161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:10:59.529547 21161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9: Bootstrap complete.
I20260924 10:10:59.538949 21161 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } }
I20260924 10:10:59.539939 21161 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf119c5899e4b3080bdf81412d3f7d9, State: Initialized, Role: FOLLOWER
I20260924 10:10:59.540580 21161 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } }
I20260924 10:10:59.540819 21161 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:10:59.540943 21161 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:10:59.541126 21161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 2 FOLLOWER]: Advancing to term 3
I20260924 10:10:59.544924 21161 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } }
I20260924 10:10:59.545408 21161 leader_election.cc:302] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf119c5899e4b3080bdf81412d3f7d9; no voters:
I20260924 10:10:59.546131 21161 leader_election.cc:288] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 10:10:59.546311 21165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 10:10:59.548261 21161 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:10:59.552898 21165 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [term 3 LEADER]: Becoming Leader. State: Replica: ddf119c5899e4b3080bdf81412d3f7d9, State: Running, Role: LEADER
I20260924 10:10:59.553574 21165 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } }
I20260924 10:10:59.561115 21166 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } } }
I20260924 10:10:59.561491 21166 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:59.561142 21167 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf119c5899e4b3080bdf81412d3f7d9. Latest consensus state: current_term: 3 leader_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf119c5899e4b3080bdf81412d3f7d9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36627 } } }
I20260924 10:10:59.562546 21167 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf119c5899e4b3080bdf81412d3f7d9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:10:59.563510 21176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:10:59.574414 21176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:10:59.577133 21176 catalog_manager.cc:1259] Loaded cluster ID: 66b602989e99440ca5b1e95c573f3fe7
I20260924 10:10:59.577442 21176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 10:10:59.579734 21176 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f33b036b9ad google::LogMessage::Fail() at ??:0
@ 0x7f33b036e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f33b036b4de google::LogMessage::Flush() at ??:0
@ 0x7f33b036ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f33c5da2512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f33c5d9f976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f33c5e27610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f33c5e27439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f33b2fb7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f33b2fba91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f33b2fba799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f33b2f95eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f33b626f6db start_thread at ??:0
@ 0x7f33af20561f clone at ??:0
W20260924 10:10:59.850458 21163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:10:59.850816 21163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:10:59.850976 21163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:10:59.858507 21163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:10:59.858665 21163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:10:59.858850 21163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:10:59.869139 21163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:37927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=36011
--tserver_master_addrs=127.6.2.254:36627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:10:59.870606 21163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:10:59.871997 21163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:10:59.885346 21203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:59.885394 21206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:10:59.885416 21204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:10:59.886315 21163 server_base.cc:1034] running on GCE node
I20260924 10:10:59.886936 21163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:10:59.887889 21163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:10:59.889112 21163 hybrid_clock.cc:648] HybridClock initialized: now 1790244659889067 us; error 57 us; skew 500 ppm
I20260924 10:10:59.889415 21163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:10:59 dist-test-slave-s3n3 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244659, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:10:59.892645 21163 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:10:59.894055 21163 webserver.cc:460] Webserver started at http://127.6.2.193:36011/ using document root <none> and password file <none>
I20260924 10:10:59.894738 21163 fs_manager.cc:359] Metadata directory not provided
I20260924 10:10:59.894886 21163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:10:59.900264 21163 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:10:59.902925 21213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:59.904245 21163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:10:59.904428 21163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "df0b97be3cae49b8985fd641f6375f87"
format_stamp: "Formatted at 2026-09-24 10:10:53 on dist-test-slave-s3n3"
server_key: "28fee85218e345f36a31c71d829ea65b"
server_key_iv: "584ca27498ffec8b40a8c72e1fb3419c"
server_key_version: "encryptionkey@0"
I20260924 10:10:59.905421 21163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:10:59.928009 21163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:10:59.932412 21163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:10:59.932701 21163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:10:59.933835 21163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:10:59.935303 21163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:10:59.935431 21163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:59.935570 21163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:10:59.935662 21163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:10:59.968640 21163 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:37927
I20260924 10:10:59.968725 21325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:37927 every 8 connection(s)
I20260924 10:10:59.970208 21163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:10:59.974891 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:59.973153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:46587)
0924 10:10:59.973845 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:59.974060 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":272,"threads_started":1}
W20260924 10:10:59.975582 21326 heartbeater.cc:644] Failed to heartbeat to 127.6.2.254:36627 (0 consecutive failures): Network error: Failed to ping master at 127.6.2.254:36627: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
I20260924 10:10:59.976434 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:59.976095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:54691)
0924 10:10:59.976246 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:59.976323 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 10:10:59.977290 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:10:59.976980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:45685)
0924 10:10:59.977113 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 10:10:59.977190 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:10:59.977546 21326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:10:59.977577 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 21163
I20260924 10:10:59.978186 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:38077
--local_ip_for_outbound_sockets=127.6.2.194
--tserver_master_addrs=127.6.2.254:36627
--webserver_port=45043
--webserver_interface=127.6.2.194
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:11:00.319463 21329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:11:00.319796 21329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:11:00.319964 21329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:11:00.327256 21329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:11:00.327421 21329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:11:00.327611 21329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:11:00.337500 21329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:38077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=45043
--tserver_master_addrs=127.6.2.254:36627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:11:00.338867 21329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:11:00.340209 21329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:11:00.353078 21337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:00.353161 21335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:00.353185 21334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:11:00.353839 21329 server_base.cc:1034] running on GCE node
I20260924 10:11:00.354423 21329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:11:00.355463 21329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:11:00.356691 21329 hybrid_clock.cc:648] HybridClock initialized: now 1790244660356608 us; error 78 us; skew 500 ppm
I20260924 10:11:00.357007 21329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:11:00 dist-test-slave-s3n3 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244660, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:11:00.360152 21329 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:11:00.361548 21329 webserver.cc:460] Webserver started at http://127.6.2.194:45043/ using document root <none> and password file <none>
I20260924 10:11:00.362178 21329 fs_manager.cc:359] Metadata directory not provided
I20260924 10:11:00.362274 21329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:11:00.367350 21329 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:11:00.369990 21344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:00.371495 21329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:11:00.371670 21329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "5689d5ac27e843f29a43c321eccaced6"
format_stamp: "Formatted at 2026-09-24 10:10:53 on dist-test-slave-s3n3"
server_key: "c205be34fe412fbcb2783337b9b2ca07"
server_key_iv: "b2bd0a33f04dd7c9c53eff6b3c41a7d5"
server_key_version: "encryptionkey@0"
I20260924 10:11:00.372632 21329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:11:00.392320 21329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:11:00.396391 21329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:11:00.396665 21329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:11:00.397656 21329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:11:00.399003 21329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:11:00.399166 21329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:00.399284 21329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:11:00.399376 21329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:00.432734 21329 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:38077
I20260924 10:11:00.432825 21456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:38077 every 8 connection(s)
I20260924 10:11:00.434268 21329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:11:00.439145 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:00.437269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:42609)
0924 10:11:00.438061 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:00.438251 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":341,"threads_started":1}
W20260924 10:11:00.439913 21457 heartbeater.cc:644] Failed to heartbeat to 127.6.2.254:36627 (0 consecutive failures): Network error: Failed to ping master at 127.6.2.254:36627: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
I20260924 10:11:00.440891 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:00.440439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:58827)
0924 10:11:00.440603 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:00.440713 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:11:00.441182 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 21329
I20260924 10:11:00.441727 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:00.441397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:56883)
0924 10:11:00.441540 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:00.441621 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 10:11:00.441808 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:43271
--local_ip_for_outbound_sockets=127.6.2.195
--tserver_master_addrs=127.6.2.254:36627
--webserver_port=36277
--webserver_interface=127.6.2.195
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:11:00.442173 21457 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:11:00.763113 21460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:11:00.763404 21460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:11:00.763583 21460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:11:00.770910 21460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:11:00.771025 21460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:11:00.771226 21460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:11:00.781410 21460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:43271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=36277
--tserver_master_addrs=127.6.2.254:36627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:11:00.782809 21460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:11:00.784168 21460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:11:00.797250 21465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:00.797322 21468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:00.797204 21466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:11:00.798763 21460 server_base.cc:1034] running on GCE node
I20260924 10:11:00.800134 21460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:11:00.801687 21460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:11:00.802886 21460 hybrid_clock.cc:648] HybridClock initialized: now 1790244660802842 us; error 41 us; skew 500 ppm
I20260924 10:11:00.803267 21460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:11:00 dist-test-slave-s3n3 krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244660, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:11:00.806442 21460 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:11:00.807968 21460 webserver.cc:460] Webserver started at http://127.6.2.195:36277/ using document root <none> and password file <none>
I20260924 10:11:00.808670 21460 fs_manager.cc:359] Metadata directory not provided
I20260924 10:11:00.808770 21460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:11:00.814196 21460 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:11:00.816942 21475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:00.818274 21460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:11:00.818446 21460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "22adf1fa898e47848eee442f6bf771c4"
format_stamp: "Formatted at 2026-09-24 10:10:54 on dist-test-slave-s3n3"
server_key: "940a40226933b0f01ce0aeeeeda7e4f1"
server_key_iv: "82ea5cacf9b5c713e2fdc3f7c1af7a93"
server_key_version: "encryptionkey@0"
I20260924 10:11:00.819504 21460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:11:00.841131 21460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:11:00.845420 21460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:11:00.845705 21460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:11:00.846768 21460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:11:00.848187 21460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:11:00.848317 21460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:00.848459 21460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:11:00.848546 21460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:00.882519 21460 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:43271
I20260924 10:11:00.882604 21587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:43271 every 8 connection(s)
I20260924 10:11:00.884052 21460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:11:00.884661 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 21460
I20260924 10:11:00.888947 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:00.887179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:38013)
0924 10:11:00.887930 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:00.888134 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":293,"threads_started":1}
W20260924 10:11:00.889659 21588 heartbeater.cc:644] Failed to heartbeat to 127.6.2.254:36627 (0 consecutive failures): Network error: Failed to ping master at 127.6.2.254:36627: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
I20260924 10:11:00.890492 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:00.890192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:37217)
0924 10:11:00.890321 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:00.890393 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 10:11:00.891418 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:00.891039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:59561)
0924 10:11:00.891225 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:00.891306 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260924 10:11:00.891673 21588 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:11:00.978849 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:00.978066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:44973)
0924 10:11:00.978519 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:00.978607 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":225,"threads_started":1}
I20260924 10:11:01.443742 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:01.442770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:60761)
0924 10:11:01.443384 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:01.443477 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":240,"threads_started":1}
I20260924 10:11:01.893325 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:01.892433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:54607)
0924 10:11:01.892989 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:01.893080 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":273,"threads_started":1}
I20260924 10:11:01.980993 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:01.980028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:44891)
0924 10:11:01.980663 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:01.980762 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":303,"threads_started":1}
I20260924 10:11:02.445438 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:02.444718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:53445)
0924 10:11:02.445182 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:02.445273 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":225,"threads_started":1}
I20260924 10:11:02.895046 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:02.894355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:33107)
0924 10:11:02.894814 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:02.894889 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":218,"threads_started":1}
I20260924 10:11:02.982988 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:02.982131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:33345)
0924 10:11:02.982696 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:02.982790 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":329,"threads_started":1}
I20260924 10:11:03.447283 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:03.446466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:51279)
0924 10:11:03.446935 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:03.447018 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":252,"threads_started":1}
I20260924 10:11:03.896802 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:03.896046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:36797)
0924 10:11:03.896517 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:03.896609 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":245,"threads_started":1}
I20260924 10:11:03.984822 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:03.984016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:36081)
0924 10:11:03.984513 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:03.984600 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":262,"threads_started":1}
I20260924 10:11:04.448997 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:04.448250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:37757)
0924 10:11:04.448709 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:04.448801 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":224,"threads_started":1}
I20260924 10:11:04.898669 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:04.897938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:40983)
0924 10:11:04.898381 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:04.898470 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":212,"threads_started":1}
I20260924 10:11:04.986558 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:04.985772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:34779)
0924 10:11:04.986299 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:04.986385 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260924 10:11:05.450897 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:05.450085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:34319)
0924 10:11:05.450581 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:05.450664 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
I20260924 10:11:05.900633 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:05.899729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:53963)
0924 10:11:05.900333 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:05.900460 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":250,"threads_started":1}
I20260924 10:11:05.988387 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:05.987573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:46149)
0924 10:11:05.988096 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:05.988212 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":294,"threads_started":1}
I20260924 10:11:06.452747 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:06.451980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:34021)
0924 10:11:06.452483 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:06.452573 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":276,"threads_started":1}
I20260924 10:11:06.902307 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:06.901572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:49103)
0924 10:11:06.902034 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:06.902142 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":246,"threads_started":1}
I20260924 10:11:06.990229 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:06.989392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:57477)
0924 10:11:06.989942 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:06.990017 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":240,"threads_started":1}
I20260924 10:11:07.454531 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:07.453711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:40237)
0924 10:11:07.454224 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:07.454309 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":227,"threads_started":1}
I20260924 10:11:07.904603 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:07.903629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:41241)
0924 10:11:07.904213 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:07.904331 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":237,"threads_started":1}
I20260924 10:11:07.992067 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:07.991358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:55249)
0924 10:11:07.991816 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:07.991903 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260924 10:11:08.456545 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:08.455775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:59173)
0924 10:11:08.456270 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:08.456364 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
I20260924 10:11:08.906482 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:08.905779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:35453)
0924 10:11:08.906218 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:08.906301 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":219,"threads_started":1}
I20260924 10:11:08.993880 21616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:08.993127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:52603)
0924 10:11:08.993615 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:08.993695 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":233,"threads_started":1}
I20260924 10:11:09.458542 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:09.457735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:33233)
0924 10:11:09.458251 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:09.458348 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":259,"threads_started":1}
I20260924 10:11:09.908344 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:09.907558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:41131)
0924 10:11:09.908088 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:09.908169 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":284,"threads_started":1}
I20260924 10:11:09.995885 21619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:09.994979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:57379)
0924 10:11:09.995586 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:09.995690 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":392,"threads_started":1}
I20260924 10:11:10.460567 21620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:10.459766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:50849)
0924 10:11:10.460309 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:10.460403 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":301,"threads_started":1}
I20260924 10:11:10.910240 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:10.909437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:48671)
0924 10:11:10.909968 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:10.910076 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":246,"threads_started":1}
I20260924 10:11:10.997678 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:10.996896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:51943)
0924 10:11:10.997359 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:10.997452 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
I20260924 10:11:11.462371 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:11.461561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:58639)
0924 10:11:11.462071 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:11.462170 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":259,"threads_started":1}
I20260924 10:11:11.912019 21624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:11.911213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:39467)
0924 10:11:11.911686 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:11.911790 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":244,"threads_started":1}
I20260924 10:11:11.999455 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:11.998685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:47739)
0924 10:11:11.999216 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:11.999302 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":284,"threads_started":1}
I20260924 10:11:12.464432 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:12.463728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:33681)
0924 10:11:12.464181 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:12.464261 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":244,"threads_started":1}
I20260924 10:11:12.913798 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:12.913054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:56055)
0924 10:11:12.913548 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:12.913640 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":272,"threads_started":1}
I20260924 10:11:13.001154 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:13.000372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:57909)
0924 10:11:13.000882 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:13.000967 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":239,"threads_started":1}
I20260924 10:11:13.466436 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:13.465608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:57921)
0924 10:11:13.466141 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:13.466234 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":271,"threads_started":1}
I20260924 10:11:13.915570 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:13.914740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:51963)
0924 10:11:13.915300 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:13.915382 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":238,"threads_started":1}
I20260924 10:11:14.002912 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:14.002154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:44619)
0924 10:11:14.002626 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:14.002721 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":254,"threads_started":1}
I20260924 10:11:14.469089 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:14.467976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:52403)
0924 10:11:14.468758 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:14.468881 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":393,"threads_started":1}
I20260924 10:11:14.917280 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:14.916552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:34187)
0924 10:11:14.917017 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:14.917096 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":235,"threads_started":1}
I20260924 10:11:15.005060 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:15.004281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:34517)
0924 10:11:15.004764 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:15.004866 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":249,"threads_started":1}
I20260924 10:11:15.471837 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:15.470465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.194:44837)
0924 10:11:15.471335 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:15.471530 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":252,"threads_started":1}
I20260924 10:11:15.896142 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 21163
I20260924 10:11:15.919600 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:15.918713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.195:40539)
0924 10:11:15.919331 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:15.919423 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":375,"threads_started":1}
I20260924 10:11:15.921948 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:15.897380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35808 (local address 127.6.2.193:37927)
0924 10:11:15.898079 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:15.898097 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:15.898142 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:11:15.900433 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:15.900444 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:15.900901 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:15.901211 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:15.902730 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:15.904094 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:15.905649 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:15.906752 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:15.906918 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:15.914992 (+ 8074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:15.915002 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:15.915022 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:15.915149 (+ 127us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:15.918339 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:15.919128 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:15.919145 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:15.919153 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:15.919330 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:15.919868 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:15.919872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:15.919875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:15.920444 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:15.920783 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:15.921235 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 10:11:15.921673 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":344,"threads_started":1}
I20260924 10:11:15.923517 21321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:11:16.010413 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 21163
I20260924 10:11:16.010478 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:16.009231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36627 (local address 127.6.2.193:54169)
0924 10:11:16.009905 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:16.010096 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":319,"threads_started":1}
I20260924 10:11:16.025817 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 21329
I20260924 10:11:16.048005 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:16.026895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.6.2.194:38077)
0924 10:11:16.027557 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:16.027575 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:16.027626 (+ 51us) server_negotiation.cc:407] Connection header received
0924 10:11:16.029204 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:16.029216 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:16.029642 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:16.029958 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:16.032090 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:16.033444 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:16.035174 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:16.035921 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:16.036078 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:16.038966 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:16.038975 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:16.038992 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:16.039033 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:16.044014 (+ 4981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:16.044902 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:16.044909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:16.044917 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:16.045281 (+ 364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:16.045782 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:16.045786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:16.045789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:16.046456 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:16.046721 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:16.047250 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 10:11:16.047696 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":309,"threads_started":1}
I20260924 10:11:16.049515 21452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:11:16.136914 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 21329
I20260924 10:11:16.155572 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 21460
I20260924 10:11:16.178833 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:16.156737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.6.2.195:43271)
0924 10:11:16.157359 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:16.157374 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:16.157416 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:11:16.159194 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:16.159209 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:16.159677 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:16.159994 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:16.162036 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:16.163493 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:16.164960 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:16.165969 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:16.166211 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:16.169837 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:16.169853 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:16.169883 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:16.169963 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:16.174616 (+ 4653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:16.175611 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:16.175619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:16.175627 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:16.175860 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:16.176529 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:16.176535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:16.176539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:16.177200 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:16.177520 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:16.178062 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 10:11:16.178523 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":306,"threads_started":1}
I20260924 10:11:16.180243 21583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:11:16.264546 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 21460
2026-09-24T10:11:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25163 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:11:16 dist-test-slave-s3n3 krb5kdc[21646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:11:16 dist-test-slave-s3n3 krb5kdc[21646](info): set up 2 sockets
Sep 24 10:11:16 dist-test-slave-s3n3 krb5kdc[21646](info): commencing operation
krb5kdc: starting...
W20260924 10:11:17.383741 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:11:17 dist-test-slave-s3n3 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:11:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:11:17.655762 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34197
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:34197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:11:18.006450 21659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:11:18.006783 21659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:11:18.006894 21659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:11:18.015081 21659 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:11:18.015202 21659 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:11:18.015240 21659 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:11:18.015286 21659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:11:18.015350 21659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:11:18.015420 21659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:11:18.015470 21659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:11:18.015523 21659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:11:18.027244 21659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:34197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:11:18.028637 21659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:11:18.030023 21659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:11:18.043938 21664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:18.043855 21667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:18.043908 21665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:11:18.044872 21659 server_base.cc:1034] running on GCE node
I20260924 10:11:18.046007 21659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:11:18.048188 21659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:11:18.049500 21659 hybrid_clock.cc:648] HybridClock initialized: now 1790244678049438 us; error 61 us; skew 500 ppm
I20260924 10:11:18.050156 21659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:11:18 dist-test-slave-s3n3 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:11:18.054328 21659 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:11:18.056239 21659 webserver.cc:460] Webserver started at http://127.6.2.254:35085/ using document root <none> and password file <none>
I20260924 10:11:18.057018 21659 fs_manager.cc:359] Metadata directory not provided
I20260924 10:11:18.057137 21659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:11:18.057500 21659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:11:18.060812 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "b6205e8b4b0547cf9d87f7e883027d1e"
format_stamp: "Formatted at 2026-09-24 10:11:18 on dist-test-slave-s3n3"
server_key: "60cb54d30ad13703676abcb157b31492"
server_key_iv: "ead131043e2cbfecc08f3ab42b052d1b"
server_key_version: "encryptionkey@0"
I20260924 10:11:18.061652 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "b6205e8b4b0547cf9d87f7e883027d1e"
format_stamp: "Formatted at 2026-09-24 10:11:18 on dist-test-slave-s3n3"
server_key: "60cb54d30ad13703676abcb157b31492"
server_key_iv: "ead131043e2cbfecc08f3ab42b052d1b"
server_key_version: "encryptionkey@0"
I20260924 10:11:18.067330 21659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 10:11:18.071412 21674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:18.073149 21659 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:11:18.073393 21659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "b6205e8b4b0547cf9d87f7e883027d1e"
format_stamp: "Formatted at 2026-09-24 10:11:18 on dist-test-slave-s3n3"
server_key: "60cb54d30ad13703676abcb157b31492"
server_key_iv: "ead131043e2cbfecc08f3ab42b052d1b"
server_key_version: "encryptionkey@0"
I20260924 10:11:18.073642 21659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:11:18.090005 21659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:11:18.094660 21659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:11:18.094940 21659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:11:18.114485 21659 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:34197
I20260924 10:11:18.114509 21725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:34197 every 8 connection(s)
I20260924 10:11:18.115923 21659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:11:18.119850 21726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:11:18.121698 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 21659
I20260924 10:11:18.121937 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:11:18.122603 6155 external_mini_cluster.cc:1428] Setting key 4ae17ef920fb1d294d40969b7d993eb8
I20260924 10:11:18.131049 21726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Bootstrap starting.
I20260924 10:11:18.134544 21726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Neither blocks nor log segments found. Creating new log.
I20260924 10:11:18.135658 21726 log.cc:824] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:11:18 dist-test-slave-s3n3 krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:11:18.138877 21726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: No bootstrap required, opened a new log
I20260924 10:11:18.148120 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:18.124629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.6.2.254:34197)
0924 10:11:18.125477 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:18.125489 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:18.125534 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:11:18.127287 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:18.127310 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:18.127701 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:18.128090 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:18.129557 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:18.130867 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:18.132379 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:18.133086 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:18.133283 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:18.138014 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:18.138021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:18.138037 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:18.138077 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:18.142353 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:18.143865 (+ 1512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:18.143902 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:18.143910 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:18.144166 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:18.144807 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:18.144811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:18.144814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:18.145459 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:18.145771 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:18.146406 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 10:11:18.146921 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":342,"threads_started":1}
I20260924 10:11:18.149322 21726 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } }
I20260924 10:11:18.149725 21726 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:11:18.149852 21726 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6205e8b4b0547cf9d87f7e883027d1e, State: Initialized, Role: FOLLOWER
I20260924 10:11:18.150413 21726 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } }
I20260924 10:11:18.150665 21726 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:11:18.150815 21726 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:11:18.151001 21726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:11:18.153314 21726 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } }
I20260924 10:11:18.153769 21726 leader_election.cc:302] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6205e8b4b0547cf9d87f7e883027d1e; no voters:
I20260924 10:11:18.154511 21726 leader_election.cc:288] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:11:18.154793 21731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:11:18.155860 21731 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 1 LEADER]: Becoming Leader. State: Replica: b6205e8b4b0547cf9d87f7e883027d1e, State: Running, Role: LEADER
I20260924 10:11:18.156486 21731 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } }
I20260924 10:11:18.156811 21726 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:11:18.159525 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } } }
I20260924 10:11:18.160315 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: This master's current role is: LEADER
I20260924 10:11:18.160175 21733 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6205e8b4b0547cf9d87f7e883027d1e. Latest consensus state: current_term: 1 leader_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } } }
I20260924 10:11:18.160576 21733 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: This master's current role is: LEADER
I20260924 10:11:18.165784 21740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:11:18.171345 21740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:11:18.182376 21740 catalog_manager.cc:1347] Generated new cluster ID: d7095e5eb7fa45a89db8ec96da5674be
I20260924 10:11:18.182510 21740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:11:18.229262 21740 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:11:18.230757 21740 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:11:18.243336 21740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Generated new TSK 0
I20260924 10:11:18.244257 21740 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:11:18.319988 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34197
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:11:18.668319 21752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:11:18.668612 21752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:11:18.668768 21752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:11:18.676282 21752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:11:18.676412 21752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:11:18.676585 21752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:11:18.686857 21752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:11:18.688261 21752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:11:18.689540 21752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:11:18.703598 21757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:18.703554 21760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:18.703572 21758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:11:18.704353 21752 server_base.cc:1034] running on GCE node
I20260924 10:11:18.705010 21752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:11:18.706043 21752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:11:18.707247 21752 hybrid_clock.cc:648] HybridClock initialized: now 1790244678707178 us; error 51 us; skew 500 ppm
I20260924 10:11:18.707617 21752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:11:18 dist-test-slave-s3n3 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:11:18.711340 21752 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:11:18.713054 21752 webserver.cc:460] Webserver started at http://127.6.2.193:44267/ using document root <none> and password file <none>
I20260924 10:11:18.713807 21752 fs_manager.cc:359] Metadata directory not provided
I20260924 10:11:18.713920 21752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:11:18.714253 21752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:11:18.717626 21752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "ddbf040cea454ed8bcffb944fcc007f0"
format_stamp: "Formatted at 2026-09-24 10:11:18 on dist-test-slave-s3n3"
server_key: "be9d8fe23b276fb565f7e98aa8a637b8"
server_key_iv: "35a7cd5bb477cbf9d05f798096c3a7d5"
server_key_version: "encryptionkey@0"
I20260924 10:11:18.718508 21752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "ddbf040cea454ed8bcffb944fcc007f0"
format_stamp: "Formatted at 2026-09-24 10:11:18 on dist-test-slave-s3n3"
server_key: "be9d8fe23b276fb565f7e98aa8a637b8"
server_key_iv: "35a7cd5bb477cbf9d05f798096c3a7d5"
server_key_version: "encryptionkey@0"
I20260924 10:11:18.724470 21752 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 10:11:18.728364 21767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:18.729851 21752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:11:18.730050 21752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "ddbf040cea454ed8bcffb944fcc007f0"
format_stamp: "Formatted at 2026-09-24 10:11:18 on dist-test-slave-s3n3"
server_key: "be9d8fe23b276fb565f7e98aa8a637b8"
server_key_iv: "35a7cd5bb477cbf9d05f798096c3a7d5"
server_key_version: "encryptionkey@0"
I20260924 10:11:18.730286 21752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:11:18.751233 21752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:11:18.756103 21752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:11:18.756425 21752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:11:18.757501 21752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:11:18.758997 21752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:11:18.759115 21752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 10:11:18.759267 21752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:11:18.759346 21752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:18.795892 21752 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:42707
I20260924 10:11:18.796119 21879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:42707 every 8 connection(s)
I20260924 10:11:18.797268 21752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:11:18.800509 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 21752
I20260924 10:11:18.800750 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:11:18.801656 6155 external_mini_cluster.cc:1428] Setting key 94b7a5c8110d459f4fddc3a0828c1d92
Sep 24 10:11:18 dist-test-slave-s3n3 krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:11:18.822837 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:18.800169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:40953 (local address 127.6.2.254:34197)
0924 10:11:18.800681 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:18.800687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:18.802231 (+ 1544us) server_negotiation.cc:407] Connection header received
0924 10:11:18.805666 (+ 3435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:18.805676 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:18.805770 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:18.805893 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:18.808168 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:18.809439 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:18.811638 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:18.812455 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:18.812634 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:18.816541 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:18.816546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:18.816549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:18.816584 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:18.819558 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:18.820369 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:18.820379 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:18.820382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:18.820506 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:18.821029 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:18.821036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:18.821039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:18.821319 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:18.821478 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:18.822213 (+ 735us) server_negotiation.cc:299] Negotiation successful
0924 10:11:18.822453 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":263,"threads_started":1}
I20260924 10:11:18.824010 21883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:18.800861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:40953)
0924 10:11:18.801652 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:18.801697 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:18.805320 (+ 3623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:18.806131 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:18.806141 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:18.806784 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:11:18.807727 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:18.807756 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:18.809724 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:18.809731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:18.811473 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:18.811491 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:18.812720 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:18.812725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:18.813027 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:18.813863 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:11:18.813912 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:11:18.816278 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:11:18.819727 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:18.819738 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:18.819757 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:18.820228 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:18.820622 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:18.820626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:18.820630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:18.820907 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:18.821450 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:11:18.821458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:11:18.821828 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 10:11:18.822107 (+ 279us) client_negotiation.cc:240] Negotiation successful
0924 10:11:18.822699 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":338,"threads_started":1}
I20260924 10:11:18.825680 21880 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34197
I20260924 10:11:18.826045 21880 heartbeater.cc:464] Registering TS with master...
I20260924 10:11:18.827083 21880 heartbeater.cc:511] Master 127.6.2.254:34197 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
I20260924 10:11:18.829641 21691 ts_manager.cc:194] Registered new tserver with Master: ddbf040cea454ed8bcffb944fcc007f0 (127.6.2.193:42707)
I20260924 10:11:18.831525 21691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:40953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:11:18.863674 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34197
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:11:19.203126 21886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:11:19.203449 21886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:11:19.203598 21886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:11:19.211218 21886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:11:19.211340 21886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:11:19.211525 21886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:11:19.221758 21886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:11:19.223182 21886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:11:19.224447 21886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:11:19.238420 21891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:19.238462 21892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:19.238484 21894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:11:19.245224 21886 server_base.cc:1034] running on GCE node
I20260924 10:11:19.246060 21886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:11:19.247299 21886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:11:19.248550 21886 hybrid_clock.cc:648] HybridClock initialized: now 1790244679248493 us; error 57 us; skew 500 ppm
I20260924 10:11:19.248930 21886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:11:19 dist-test-slave-s3n3 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244679, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:11:19.252365 21886 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:11:19.254136 21886 webserver.cc:460] Webserver started at http://127.6.2.194:41727/ using document root <none> and password file <none>
I20260924 10:11:19.254889 21886 fs_manager.cc:359] Metadata directory not provided
I20260924 10:11:19.255030 21886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:11:19.255380 21886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:11:19.258577 21886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "103da8104d9f496fb4998c589a5db906"
format_stamp: "Formatted at 2026-09-24 10:11:19 on dist-test-slave-s3n3"
server_key: "8806cf784631be0068056fc76852f253"
server_key_iv: "d9ab185082fa452d1da1492f31f53b3d"
server_key_version: "encryptionkey@0"
I20260924 10:11:19.259431 21886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "103da8104d9f496fb4998c589a5db906"
format_stamp: "Formatted at 2026-09-24 10:11:19 on dist-test-slave-s3n3"
server_key: "8806cf784631be0068056fc76852f253"
server_key_iv: "d9ab185082fa452d1da1492f31f53b3d"
server_key_version: "encryptionkey@0"
I20260924 10:11:19.264863 21886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 10:11:19.269483 21901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:19.271557 21886 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:11:19.271867 21886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "103da8104d9f496fb4998c589a5db906"
format_stamp: "Formatted at 2026-09-24 10:11:19 on dist-test-slave-s3n3"
server_key: "8806cf784631be0068056fc76852f253"
server_key_iv: "d9ab185082fa452d1da1492f31f53b3d"
server_key_version: "encryptionkey@0"
I20260924 10:11:19.272107 21886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:11:19.313357 21886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:11:19.317986 21886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:11:19.318305 21886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:11:19.319460 21886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:11:19.320911 21886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:11:19.321012 21886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:19.321151 21886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:11:19.321218 21886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:19.354929 21886 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:40735
I20260924 10:11:19.355014 22013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:40735 every 8 connection(s)
I20260924 10:11:19.356240 21886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:11:19.359982 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 21886
I20260924 10:11:19.360195 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:11:19.360811 6155 external_mini_cluster.cc:1428] Setting key a22ce5526c1b942a422f45ed4278d879
Sep 24 10:11:19 dist-test-slave-s3n3 krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244679, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:11:19.377759 22016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:19.358864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:47007 (local address 127.6.2.254:34197)
0924 10:11:19.359376 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:19.359383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:19.360485 (+ 1102us) server_negotiation.cc:407] Connection header received
0924 10:11:19.362951 (+ 2466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:19.362961 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:19.363047 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:19.363190 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:19.364811 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.365693 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.367285 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.367731 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.367867 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:19.371340 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:19.371345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:19.371348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:19.371381 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:19.374073 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:19.374856 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:19.374861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:19.374865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:19.375003 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:19.375498 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:19.375508 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:19.375541 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:19.375827 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:19.375979 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:19.377042 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0924 10:11:19.377359 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":310,"threads_started":1}
I20260924 10:11:19.378810 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:19.359481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:47007)
0924 10:11:19.360300 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:19.360347 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:19.362649 (+ 2302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:19.363424 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:19.363432 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:19.363975 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:11:19.364643 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:19.364662 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.365843 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.365847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:19.367149 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:19.367161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.367962 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.367969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:19.368199 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:19.368896 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:11:19.368920 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:11:19.371099 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:11:19.374230 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:19.374240 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:19.374257 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:19.374725 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:19.375144 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:19.375149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:19.375153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:19.375377 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:19.375955 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:11:19.375963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:11:19.376535 (+ 572us) client_negotiation.cc:769] Sending connection context
0924 10:11:19.376925 (+ 390us) client_negotiation.cc:240] Negotiation successful
0924 10:11:19.377487 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":323,"threads_started":1}
I20260924 10:11:19.380406 22014 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34197
I20260924 10:11:19.380738 22014 heartbeater.cc:464] Registering TS with master...
I20260924 10:11:19.381541 22014 heartbeater.cc:511] Master 127.6.2.254:34197 requested a full tablet report, sending...
I20260924 10:11:19.383546 21691 ts_manager.cc:194] Registered new tserver with Master: 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735)
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:11:19.384461 21691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:47007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:11:19.415499 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34197
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:11:19.755740 22020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:11:19.756038 22020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:11:19.756208 22020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:11:19.763819 22020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:11:19.763942 22020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:11:19.764173 22020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:11:19.774322 22020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:11:19.775756 22020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:11:19.777155 22020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:11:19.790690 22025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:19.790647 22028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:19.790731 22026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:11:19.791563 22020 server_base.cc:1034] running on GCE node
I20260924 10:11:19.792201 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:11:19.793251 22020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:11:19.794504 22020 hybrid_clock.cc:648] HybridClock initialized: now 1790244679794436 us; error 63 us; skew 500 ppm
I20260924 10:11:19.794837 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:11:19 dist-test-slave-s3n3 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244679, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:11:19.798182 22020 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:11:19.799857 22020 webserver.cc:460] Webserver started at http://127.6.2.195:44749/ using document root <none> and password file <none>
I20260924 10:11:19.800532 22020 fs_manager.cc:359] Metadata directory not provided
I20260924 10:11:19.800655 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:11:19.800958 22020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:11:19.803936 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "216bcbd68e2c489991d5cc631c7c213c"
format_stamp: "Formatted at 2026-09-24 10:11:19 on dist-test-slave-s3n3"
server_key: "33759a5f759b6063ce32b8896eebddbf"
server_key_iv: "7a4e16f0c7c170897a668925480bd5b5"
server_key_version: "encryptionkey@0"
I20260924 10:11:19.804692 22020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "216bcbd68e2c489991d5cc631c7c213c"
format_stamp: "Formatted at 2026-09-24 10:11:19 on dist-test-slave-s3n3"
server_key: "33759a5f759b6063ce32b8896eebddbf"
server_key_iv: "7a4e16f0c7c170897a668925480bd5b5"
server_key_version: "encryptionkey@0"
I20260924 10:11:19.811157 22020 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 10:11:19.814720 22035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:19.816537 22020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:11:19.816753 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "216bcbd68e2c489991d5cc631c7c213c"
format_stamp: "Formatted at 2026-09-24 10:11:19 on dist-test-slave-s3n3"
server_key: "33759a5f759b6063ce32b8896eebddbf"
server_key_iv: "7a4e16f0c7c170897a668925480bd5b5"
server_key_version: "encryptionkey@0"
I20260924 10:11:19.816987 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:11:19.834589 22020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:11:19.835011 21880 heartbeater.cc:503] Master 127.6.2.254:34197 was elected leader, sending a full tablet report...
I20260924 10:11:19.839465 22020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:11:19.839708 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:11:19.840801 22020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:11:19.842176 22020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:11:19.842278 22020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:19.842423 22020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:11:19.842491 22020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:19.878690 22020 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:43631
I20260924 10:11:19.878712 22147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:43631 every 8 connection(s)
I20260924 10:11:19.880030 22020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:11:19.880525 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 22020
I20260924 10:11:19.880725 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:11:19.881345 6155 external_mini_cluster.cc:1428] Setting key 195fb0755fb14a49e41892a344c1f795
Sep 24 10:11:19 dist-test-slave-s3n3 krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244679, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:11:19.900738 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:19.882662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:46795 (local address 127.6.2.254:34197)
0924 10:11:19.883206 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:19.883212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:19.884178 (+ 966us) server_negotiation.cc:407] Connection header received
0924 10:11:19.886084 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:19.886089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:19.886174 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:19.886293 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:19.887851 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.888618 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.890125 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.890541 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.890670 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:19.894362 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:19.894373 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:19.894376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:19.894408 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:19.897555 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:19.898312 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:19.898330 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:19.898333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:19.898448 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:19.898893 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:19.898898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:19.898901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:19.899241 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:19.899400 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:19.900102 (+ 702us) server_negotiation.cc:299] Negotiation successful
0924 10:11:19.900366 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":253,"threads_started":1}
I20260924 10:11:19.901659 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:19.883317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:46795)
0924 10:11:19.884034 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:19.884073 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:19.885812 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:19.886535 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:19.886543 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:19.887114 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:11:19.887685 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:19.887702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.888769 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.888779 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:19.889984 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:19.889996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.890757 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.890764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:19.891016 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:19.891723 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:11:19.891745 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:11:19.894135 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:11:19.897707 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:19.897716 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:19.897734 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:19.898184 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:19.898562 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:19.898567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:19.898570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:19.898773 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:19.899362 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:11:19.899370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:11:19.899721 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 10:11:19.899985 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 10:11:19.900556 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":310,"threads_started":1}
I20260924 10:11:19.903574 22148 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34197
I20260924 10:11:19.904007 22148 heartbeater.cc:464] Registering TS with master...
I20260924 10:11:19.905014 22148 heartbeater.cc:511] Master 127.6.2.254:34197 requested a full tablet report, sending...
I20260924 10:11:19.906584 21691 ts_manager.cc:194] Registered new tserver with Master: 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631)
I20260924 10:11:19.907577 21691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:46795
I20260924 10:11:19.909759 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:11:19.927714 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:19.914350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45818 (local address 127.6.2.254:34197)
0924 10:11:19.914556 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:19.914565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:19.914976 (+ 411us) server_negotiation.cc:407] Connection header received
0924 10:11:19.915224 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:19.915230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:19.915312 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:19.915415 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:19.916859 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.917578 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.918956 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.919474 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.919604 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:19.922155 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:19.922162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:19.922166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:19.922212 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:19.925072 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:19.925734 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:19.925740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:19.925744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:19.925872 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:19.926235 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:19.926243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:19.926246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:19.926575 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:19.926824 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:19.927159 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 10:11:19.927391 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:11:19.931778 21691 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:11:19.938851 21691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:11:19.982266 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:19.951221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:34418)
0924 10:11:19.967054 (+ 15833us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:19.967128 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:19.967361 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:19.973851 (+ 6490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:19.973865 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:19.973906 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:11:19.974444 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:19.974456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.976665 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.976673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:19.979292 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:19.979304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.981121 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.981127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:19.981319 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:19.981375 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 10:11:19.981518 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 10:11:19.981746 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15551,"mutex_wait_us":116,"thread_start_us":15340,"threads_started":1}
I20260924 10:11:19.982828 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:19.951752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:43631 (local address 127.0.0.1:48838)
0924 10:11:19.974212 (+ 22460us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:19.974230 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:19.974457 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:19.975368 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:19.975372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:19.975397 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:11:19.975792 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:19.975804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.977915 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.977919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:19.980199 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:19.980209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.981872 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.981880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:19.982058 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:19.982142 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 10:11:19.982250 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 10:11:19.982393 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22336,"mutex_wait_us":35,"thread_start_us":21359,"threads_started":1}
I20260924 10:11:19.982970 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:19.951569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:40735 (local address 127.0.0.1:39244)
0924 10:11:19.973449 (+ 21880us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:19.973475 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:19.973605 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:19.974516 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:19.974521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:19.974542 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:11:19.974959 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:19.974972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.976888 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.976898 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:19.979628 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:19.979642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.982053 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.982059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:19.982369 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:19.982407 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 10:11:19.982489 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 10:11:19.982611 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21698,"mutex_wait_us":26,"thread_start_us":22326,"threads_started":1}
I20260924 10:11:19.983338 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:19.951535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34418 (local address 127.6.2.193:42707)
0924 10:11:19.972553 (+ 21018us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:19.972565 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:19.972605 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:11:19.972721 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:19.972733 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:19.973055 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:19.974645 (+ 1590us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:11:19.974921 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.976526 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.979455 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.980914 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.981060 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:19.981308 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:19.982741 (+ 1433us) server_negotiation.cc:299] Negotiation successful
0924 10:11:19.982960 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20811,"thread_start_us":20615,"threads_started":1}
I20260924 10:11:19.983665 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:19.952372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.6.2.195:43631)
0924 10:11:19.972908 (+ 20536us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:19.972916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:19.974342 (+ 1426us) server_negotiation.cc:407] Connection header received
0924 10:11:19.974652 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:19.974662 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:19.974861 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:19.975023 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:11:19.975957 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.977735 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.980347 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.981432 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.981544 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:19.981739 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:19.983139 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0924 10:11:19.983316 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20375,"thread_start_us":20224,"threads_started":1}
I20260924 10:11:19.983840 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:19.952169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.6.2.194:40735)
0924 10:11:19.973903 (+ 21734us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:19.973912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:19.973943 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:11:19.974045 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:19.974052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:19.974289 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:19.975203 (+ 914us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:11:19.975433 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.976750 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.979795 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:19.981593 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:19.981725 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:19.981899 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:19.983336 (+ 1437us) server_negotiation.cc:299] Negotiation successful
0924 10:11:19.983538 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21519,"thread_start_us":21364,"threads_started":1}
I20260924 10:11:19.987501 21815 tablet_service.cc:1467] Processing CreateTablet for tablet dd5233a9e6464965850929afcd081086 (DEFAULT_TABLE table=test-table [id=f98e912b39a3468e833cd6213119ddb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:11:19.987501 21949 tablet_service.cc:1467] Processing CreateTablet for tablet dd5233a9e6464965850929afcd081086 (DEFAULT_TABLE table=test-table [id=f98e912b39a3468e833cd6213119ddb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:11:19.988350 22083 tablet_service.cc:1467] Processing CreateTablet for tablet dd5233a9e6464965850929afcd081086 (DEFAULT_TABLE table=test-table [id=f98e912b39a3468e833cd6213119ddb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:11:19.988978 21815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5233a9e6464965850929afcd081086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:11:19.988978 21949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5233a9e6464965850929afcd081086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:11:19.989467 22083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5233a9e6464965850929afcd081086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:11:20.000916 22167 tablet_bootstrap.cc:492] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c: Bootstrap starting.
I20260924 10:11:20.003572 22168 tablet_bootstrap.cc:492] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906: Bootstrap starting.
I20260924 10:11:20.004354 22167 tablet_bootstrap.cc:654] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c: Neither blocks nor log segments found. Creating new log.
I20260924 10:11:20.004483 22169 tablet_bootstrap.cc:492] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Bootstrap starting.
I20260924 10:11:20.006584 22167 log.cc:824] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:11:20.007855 22168 tablet_bootstrap.cc:654] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906: Neither blocks nor log segments found. Creating new log.
I20260924 10:11:20.008718 22169 tablet_bootstrap.cc:654] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Neither blocks nor log segments found. Creating new log.
I20260924 10:11:20.008977 22168 log.cc:824] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906: Log is configured to *not* fsync() on all Append() calls
I20260924 10:11:20.009924 22169 log.cc:824] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:11:20.010610 22167 tablet_bootstrap.cc:492] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c: No bootstrap required, opened a new log
I20260924 10:11:20.010911 22167 ts_tablet_manager.cc:1397] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 10:11:20.012581 22168 tablet_bootstrap.cc:492] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906: No bootstrap required, opened a new log
I20260924 10:11:20.012921 22168 ts_tablet_manager.cc:1397] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260924 10:11:20.013948 22169 tablet_bootstrap.cc:492] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: No bootstrap required, opened a new log
I20260924 10:11:20.014289 22169 ts_tablet_manager.cc:1397] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260924 10:11:20.021934 22169 raft_consensus.cc:348] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } } peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } }
I20260924 10:11:20.022354 22169 raft_consensus.cc:374] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:11:20.022446 22169 raft_consensus.cc:729] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbf040cea454ed8bcffb944fcc007f0, State: Initialized, Role: FOLLOWER
I20260924 10:11:20.022346 22167 raft_consensus.cc:348] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } } peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } }
I20260924 10:11:20.022758 22167 raft_consensus.cc:374] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:11:20.022886 22167 raft_consensus.cc:729] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216bcbd68e2c489991d5cc631c7c213c, State: Initialized, Role: FOLLOWER
I20260924 10:11:20.023233 22169 consensus_queue.cc:260] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } } peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } }
I20260924 10:11:20.023468 22167 consensus_queue.cc:260] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } } peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } }
I20260924 10:11:20.023903 22168 raft_consensus.cc:348] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } } peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } }
I20260924 10:11:20.024524 22168 raft_consensus.cc:374] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:11:20.024616 22168 raft_consensus.cc:729] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103da8104d9f496fb4998c589a5db906, State: Initialized, Role: FOLLOWER
I20260924 10:11:20.025327 22169 ts_tablet_manager.cc:1428] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 10:11:20.025161 22168 consensus_queue.cc:260] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } } peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } }
I20260924 10:11:20.026326 22014 heartbeater.cc:503] Master 127.6.2.254:34197 was elected leader, sending a full tablet report...
I20260924 10:11:20.026998 22168 ts_tablet_manager.cc:1428] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260924 10:11:20.027954 22148 heartbeater.cc:503] Master 127.6.2.254:34197 was elected leader, sending a full tablet report...
I20260924 10:11:20.028554 22167 ts_tablet_manager.cc:1428] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
W20260924 10:11:20.050019 21881 tablet.cc:2369] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:11:20.108312 22015 tablet.cc:2369] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:11:20.131866 22149 tablet.cc:2369] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:11:20.261415 22173 raft_consensus.cc:482] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:11:20.261732 22173 raft_consensus.cc:504] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } } peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } }
I20260924 10:11:20.263441 22173 leader_election.cc:288] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735), 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631)
I20260924 10:11:20.270422 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.263887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:50357 (local address 127.6.2.195:43631)
0924 10:11:20.264102 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:20.264108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:20.265008 (+ 900us) server_negotiation.cc:407] Connection header received
0924 10:11:20.265235 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:20.265239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:20.265332 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:20.265418 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:11:20.266064 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.266774 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.268653 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.269460 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.269558 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:20.269694 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:20.270020 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 10:11:20.270158 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 10:11:20.270447 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.264008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:52361 (local address 127.6.2.194:40735)
0924 10:11:20.264231 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:20.264238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:20.264897 (+ 659us) server_negotiation.cc:407] Connection header received
0924 10:11:20.265148 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:20.265152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:20.265215 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:20.265296 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:11:20.265961 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.266713 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.268711 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.269491 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.269660 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:20.269754 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:20.270061 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 10:11:20.270187 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:11:20.270533 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.263898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:43631 (local address 127.6.2.193:50357)
0924 10:11:20.264907 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:20.264923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:20.265024 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:20.265457 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:20.265462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:20.265508 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:11:20.265890 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:20.265905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.266926 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.266933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:20.268504 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:20.268519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.269693 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.269700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:20.269866 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:20.269891 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:11:20.269967 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 10:11:20.270114 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":2,"thread_start_us":393,"threads_started":1}
I20260924 10:11:20.271297 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.263823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:40735 (local address 127.6.2.193:52361)
0924 10:11:20.264775 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:20.264795 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:20.264972 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:20.265350 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:20.265356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:20.265386 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:11:20.265806 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:20.265819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.266875 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.266882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:20.268529 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:20.268541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.269751 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.269756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:20.269888 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:20.269911 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:11:20.270315 (+ 404us) client_negotiation.cc:240] Negotiation successful
0924 10:11:20.270461 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":86,"thread_start_us":688,"threads_started":1}
I20260924 10:11:20.271567 22103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5233a9e6464965850929afcd081086" candidate_uuid: "ddbf040cea454ed8bcffb944fcc007f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216bcbd68e2c489991d5cc631c7c213c" is_pre_election: true
I20260924 10:11:20.272146 22103 raft_consensus.cc:2454] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbf040cea454ed8bcffb944fcc007f0 in term 0.
I20260924 10:11:20.272848 21771 leader_election.cc:302] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216bcbd68e2c489991d5cc631c7c213c, ddbf040cea454ed8bcffb944fcc007f0; no voters:
I20260924 10:11:20.272949 21969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5233a9e6464965850929afcd081086" candidate_uuid: "ddbf040cea454ed8bcffb944fcc007f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103da8104d9f496fb4998c589a5db906" is_pre_election: true
I20260924 10:11:20.273262 22173 raft_consensus.cc:2781] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:11:20.273432 22173 raft_consensus.cc:482] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:11:20.273553 22173 raft_consensus.cc:3037] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:11:20.273550 21969 raft_consensus.cc:2454] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbf040cea454ed8bcffb944fcc007f0 in term 0.
I20260924 10:11:20.276161 22173 raft_consensus.cc:504] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } } peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } }
I20260924 10:11:20.276852 22173 leader_election.cc:288] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 1 election: Requested vote from peers 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735), 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631)
I20260924 10:11:20.277249 21969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5233a9e6464965850929afcd081086" candidate_uuid: "ddbf040cea454ed8bcffb944fcc007f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103da8104d9f496fb4998c589a5db906"
I20260924 10:11:20.277339 22103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5233a9e6464965850929afcd081086" candidate_uuid: "ddbf040cea454ed8bcffb944fcc007f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216bcbd68e2c489991d5cc631c7c213c"
I20260924 10:11:20.277437 21969 raft_consensus.cc:3037] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:11:20.277580 22103 raft_consensus.cc:3037] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:11:20.279896 22103 raft_consensus.cc:2454] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbf040cea454ed8bcffb944fcc007f0 in term 1.
I20260924 10:11:20.279899 21969 raft_consensus.cc:2454] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbf040cea454ed8bcffb944fcc007f0 in term 1.
I20260924 10:11:20.280290 21771 leader_election.cc:302] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216bcbd68e2c489991d5cc631c7c213c, ddbf040cea454ed8bcffb944fcc007f0; no voters:
I20260924 10:11:20.280580 22173 raft_consensus.cc:2781] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:11:20.281054 22173 raft_consensus.cc:686] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 LEADER]: Becoming Leader. State: Replica: ddbf040cea454ed8bcffb944fcc007f0, State: Running, Role: LEADER
I20260924 10:11:20.281512 22173 consensus_queue.cc:237] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } } peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } }
I20260924 10:11:20.286190 21691 catalog_manager.cc:5495] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddbf040cea454ed8bcffb944fcc007f0 (127.6.2.193). New cstate: current_term: 1 leader_uuid: "ddbf040cea454ed8bcffb944fcc007f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } health_report { overall_health: UNKNOWN } } }
I20260924 10:11:20.354161 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.349238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.6.2.193:42707)
0924 10:11:20.349427 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:20.349431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:20.349447 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:11:20.349576 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:20.349580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:20.349658 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:20.349788 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:11:20.350110 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.350832 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.352183 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.352589 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.352684 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:20.353173 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:11:20.353672 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:11:20.353809 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:20.353849 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 10:11:20.353914 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:11:20.363018 21969 raft_consensus.cc:1261] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906 [term 1 FOLLOWER]: Refusing update from remote peer ddbf040cea454ed8bcffb944fcc007f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:11:20.363133 22103 raft_consensus.cc:1261] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c [term 1 FOLLOWER]: Refusing update from remote peer ddbf040cea454ed8bcffb944fcc007f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:11:20.364058 22178 consensus_queue.cc:1035] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:11:20.364442 22173 consensus_queue.cc:1035] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:11:20.375983 22181 mvcc.cc:205] Tried to move back new op lower bound from 7332842210756726784 to 7332842210436575232. Current Snapshot: MvccSnapshot[applied={T|T < 7332842210756726784}]
I20260924 10:11:20.389935 21691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:11:20.390199 21691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:11:20.393950 21691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Sending DeleteTablet for 3 replicas of tablet dd5233a9e6464965850929afcd081086
I20260924 10:11:20.395152 21815 tablet_service.cc:1514] Processing DeleteTablet for tablet dd5233a9e6464965850929afcd081086 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:11:20 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:34418
I20260924 10:11:20.395224 21949 tablet_service.cc:1514] Processing DeleteTablet for tablet dd5233a9e6464965850929afcd081086 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:11:20 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:39244
I20260924 10:11:20.395572 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 21752
I20260924 10:11:20.395496 22083 tablet_service.cc:1514] Processing DeleteTablet for tablet dd5233a9e6464965850929afcd081086 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:11:20 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:48838
I20260924 10:11:20.416198 22191 tablet_replica.cc:330] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906: stopping tablet replica
I20260924 10:11:20.416384 22192 tablet_replica.cc:330] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c: stopping tablet replica
I20260924 10:11:20.416898 22192 raft_consensus.cc:2229] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:11:20.416929 22191 raft_consensus.cc:2229] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:11:20.417339 22190 tablet_replica.cc:330] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: stopping tablet replica
I20260924 10:11:20.417478 22192 raft_consensus.cc:2258] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:11:20.417735 22191 raft_consensus.cc:2258] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 10:11:20 dist-test-slave-s3n3 krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:11:20.419873 22192 ts_tablet_manager.cc:1905] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:11:20.419880 22191 ts_tablet_manager.cc:1905] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:11:20.422036 22190 raft_consensus.cc:2229] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:11:20.427050 22191 ts_tablet_manager.cc:1918] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:11:20.427309 22191 log.cc:1196] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/dd5233a9e6464965850929afcd081086
I20260924 10:11:20.427779 22192 ts_tablet_manager.cc:1918] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:11:20.427994 22191 ts_tablet_manager.cc:1939] T dd5233a9e6464965850929afcd081086 P 103da8104d9f496fb4998c589a5db906: Deleting consensus metadata
I20260924 10:11:20.428218 22192 log.cc:1196] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/dd5233a9e6464965850929afcd081086
I20260924 10:11:20.428694 22192 ts_tablet_manager.cc:1939] T dd5233a9e6464965850929afcd081086 P 216bcbd68e2c489991d5cc631c7c213c: Deleting consensus metadata
I20260924 10:11:20.428810 22190 raft_consensus.cc:2258] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:11:20.430008 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.397065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.6.2.193:42707)
0924 10:11:20.397306 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:20.397315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:20.406553 (+ 9238us) server_negotiation.cc:407] Connection header received
0924 10:11:20.407046 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:20.407052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:20.407187 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:20.407304 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:20.410079 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.411645 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.413896 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.414611 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.414712 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:20.421039 (+ 6327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:20.421057 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:20.421075 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:20.421125 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:20.425394 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:20.426372 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:20.426380 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:20.426384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:20.426556 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:20.427508 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:20.427514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:20.427517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:20.427974 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:20.428275 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:20.429361 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0924 10:11:20.429621 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:11:20.430220 21676 catalog_manager.cc:4841] TS 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735): tablet dd5233a9e6464965850929afcd081086 (table test-table [id=f98e912b39a3468e833cd6213119ddb0]) successfully deleted
I20260924 10:11:20.430537 21678 catalog_manager.cc:4841] TS 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631): tablet dd5233a9e6464965850929afcd081086 (table test-table [id=f98e912b39a3468e833cd6213119ddb0]) successfully deleted
I20260924 10:11:20.431602 21875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:11:20.432328 22190 ts_tablet_manager.cc:1905] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:11:20.530844 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 21752
I20260924 10:11:20.550338 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 21886
W20260924 10:11:20.550390 21675 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.2.193:42707 (error 108)
I20260924 10:11:20.552369 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.551687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:34448)
0924 10:11:20.552117 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:20.552211 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":218,"threads_started":1}
W20260924 10:11:20.552562 21675 catalog_manager.cc:4584] TS ddbf040cea454ed8bcffb944fcc007f0 (127.6.2.193:42707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd5233a9e6464965850929afcd081086: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Sep 24 10:11:20 dist-test-slave-s3n3 krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:11:20.566409 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.551199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.6.2.194:40735)
0924 10:11:20.551396 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:20.551405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:20.551456 (+ 51us) server_negotiation.cc:407] Connection header received
0924 10:11:20.551671 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:20.551675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:20.551747 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:20.551865 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:20.553305 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.553991 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.555401 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.555790 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.555920 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:20.560341 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:20.560356 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:20.560373 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:20.560409 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:20.563513 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:20.564245 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:20.564253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:20.564255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:20.564410 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:20.564796 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:20.564802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:20.564805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:20.565184 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:20.565490 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:20.565785 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 10:11:20.566050 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:11:20.567137 22009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:11:20.606269 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.605800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:34450)
0924 10:11:20.605991 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:20.606095 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 10:11:20.607528 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.607016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:34464)
0924 10:11:20.607274 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:20.607374 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 10:11:20.658080 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 21886
I20260924 10:11:20.668196 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.667575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:34480)
0924 10:11:20.667856 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:20.667981 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 10:11:20.669450 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.668958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:34492)
0924 10:11:20.669142 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:20.669241 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:11:20.676548 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 22020
Sep 24 10:11:20 dist-test-slave-s3n3 krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:11:20.695629 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.677629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.6.2.195:43631)
0924 10:11:20.677866 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:20.677872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:20.677896 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:11:20.677981 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:20.677986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:20.678072 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:20.678233 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:20.680440 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.681619 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.683248 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:20.683666 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:20.683783 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:20.688882 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:20.688897 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:20.688917 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:20.688958 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:20.692226 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:20.693407 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:20.693414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:20.693416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:20.693599 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:20.693959 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:20.693962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:20.693964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:20.694319 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:20.694579 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:20.695187 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 10:11:20.695375 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 10:11:20.696354 22143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:11:20.750833 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.750310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:34496)
0924 10:11:20.750553 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:20.750662 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 10:11:20.752029 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:20.751544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:34502)
0924 10:11:20.751787 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:20.751893 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 10:11:20.788405 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 22020
I20260924 10:11:20.806035 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 21659
I20260924 10:11:20.807283 21721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:11:20.922700 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 21659
I20260924 10:11:40.945353 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34197
--webserver_interface=127.6.2.254
--webserver_port=35085
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:34197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:11:41.280776 22200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:11:41.281139 22200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:11:41.281313 22200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:11:41.288787 22200 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:11:41.288942 22200 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:11:41.289019 22200 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:11:41.289099 22200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:11:41.289179 22200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:11:41.289304 22200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:11:41.289393 22200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:11:41.289474 22200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:11:41.300158 22200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:34197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=35085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:11:41.301537 22200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:11:41.302881 22200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:11:41.327286 22207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:41.327422 22209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:41.327462 22206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:11:41.327806 22200 server_base.cc:1034] running on GCE node
I20260924 10:11:41.329041 22200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:11:41.330732 22200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:11:41.332007 22200 hybrid_clock.cc:648] HybridClock initialized: now 1790244701331913 us; error 72 us; skew 500 ppm
I20260924 10:11:41.332460 22200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:11:41 dist-test-slave-s3n3 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:11:41.336194 22200 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:11:41.338093 22200 webserver.cc:460] Webserver started at http://127.6.2.254:35085/ using document root <none> and password file <none>
I20260924 10:11:41.338935 22200 fs_manager.cc:359] Metadata directory not provided
I20260924 10:11:41.339094 22200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:11:41.345204 22200 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 10:11:41.348358 22216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:41.350168 22200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:11:41.350337 22200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "b6205e8b4b0547cf9d87f7e883027d1e"
format_stamp: "Formatted at 2026-09-24 10:11:18 on dist-test-slave-s3n3"
server_key: "60cb54d30ad13703676abcb157b31492"
server_key_iv: "ead131043e2cbfecc08f3ab42b052d1b"
server_key_version: "encryptionkey@0"
I20260924 10:11:41.351482 22200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:11:41.377703 22200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:11:41.382584 22200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:11:41.382843 22200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:11:41.402839 22200 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:34197
I20260924 10:11:41.402868 22267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:34197 every 8 connection(s)
I20260924 10:11:41.404443 22200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:11:41.410609 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 22200
I20260924 10:11:41.411711 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:42707
--local_ip_for_outbound_sockets=127.6.2.193
--tserver_master_addrs=127.6.2.254:34197
--webserver_port=44267
--webserver_interface=127.6.2.193
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:11:41.412547 22268 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:11:41.418053 22268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Bootstrap starting.
I20260924 10:11:41.438861 22268 log.cc:824] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:11:41.451161 22268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:11:41.451750 22268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Bootstrap complete.
I20260924 10:11:41.461278 22268 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } }
I20260924 10:11:41.462122 22268 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6205e8b4b0547cf9d87f7e883027d1e, State: Initialized, Role: FOLLOWER
I20260924 10:11:41.462761 22268 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } }
I20260924 10:11:41.463009 22268 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:11:41.463150 22268 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:11:41.463310 22268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:11:41.467309 22268 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } }
I20260924 10:11:41.467901 22268 leader_election.cc:302] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6205e8b4b0547cf9d87f7e883027d1e; no voters:
I20260924 10:11:41.468601 22268 leader_election.cc:288] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:11:41.468731 22272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:11:41.469980 22272 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 2 LEADER]: Becoming Leader. State: Replica: b6205e8b4b0547cf9d87f7e883027d1e, State: Running, Role: LEADER
I20260924 10:11:41.470628 22272 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } }
I20260924 10:11:41.470974 22268 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:11:41.473244 22274 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6205e8b4b0547cf9d87f7e883027d1e. Latest consensus state: current_term: 2 leader_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } } }
I20260924 10:11:41.473611 22274 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: This master's current role is: LEADER
I20260924 10:11:41.473363 22273 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } } }
I20260924 10:11:41.473798 22273 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: This master's current role is: LEADER
I20260924 10:11:41.477902 22279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:11:41.487748 22279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:11:41.489763 22279 catalog_manager.cc:1259] Loaded cluster ID: d7095e5eb7fa45a89db8ec96da5674be
I20260924 10:11:41.489974 22279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:11:41.495187 22279 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:11:41.497354 22279 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Loaded TSK: 0
I20260924 10:11:41.499694 22289 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:11:41.523676 22279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Generated new TSK 1
I20260924 10:11:41.524127 22279 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:11:41.783398 22270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:11:41.783703 22270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:11:41.783872 22270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:11:41.791785 22270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:11:41.791901 22270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:11:41.792073 22270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:11:41.802385 22270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:42707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=44267
--tserver_master_addrs=127.6.2.254:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:11:41.803910 22270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:11:41.805284 22270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:11:41.821058 22295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:41.821058 22296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:41.821885 22298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:11:41.822299 22270 server_base.cc:1034] running on GCE node
I20260924 10:11:41.822916 22270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:11:41.824105 22270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:11:41.825377 22270 hybrid_clock.cc:648] HybridClock initialized: now 1790244701825315 us; error 44 us; skew 500 ppm
I20260924 10:11:41.825740 22270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:11:41 dist-test-slave-s3n3 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:11:41.830483 22270 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:11:41.832576 22270 webserver.cc:460] Webserver started at http://127.6.2.193:44267/ using document root <none> and password file <none>
I20260924 10:11:41.833367 22270 fs_manager.cc:359] Metadata directory not provided
I20260924 10:11:41.833537 22270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:11:41.839870 22270 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 10:11:41.843319 22305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:41.844901 22270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:11:41.845091 22270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "ddbf040cea454ed8bcffb944fcc007f0"
format_stamp: "Formatted at 2026-09-24 10:11:18 on dist-test-slave-s3n3"
server_key: "be9d8fe23b276fb565f7e98aa8a637b8"
server_key_iv: "35a7cd5bb477cbf9d05f798096c3a7d5"
server_key_version: "encryptionkey@0"
I20260924 10:11:41.846144 22270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:11:41.880779 22270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:11:41.885031 22270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:11:41.885326 22270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:11:41.886391 22270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:11:41.888375 22312 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 10:11:41.894853 22270 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:11:41.895005 22270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260924 10:11:41.895184 22270 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:11:41.898379 22270 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:11:41.898491 22270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260924 10:11:41.898890 22312 tablet_bootstrap.cc:492] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Bootstrap starting.
I20260924 10:11:41.908142 22312 log.cc:824] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:11:41.926998 22312 tablet_bootstrap.cc:492] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:11:41.927937 22312 tablet_bootstrap.cc:492] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Bootstrap complete.
I20260924 10:11:41.928483 22312 ts_tablet_manager.cc:1397] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.006s
I20260924 10:11:41.936520 22270 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:42707
I20260924 10:11:41.936611 22419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:42707 every 8 connection(s)
I20260924 10:11:41.937920 22270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:11:41.940184 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 22270
I20260924 10:11:41.940943 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:40735
--local_ip_for_outbound_sockets=127.6.2.194
--tserver_master_addrs=127.6.2.254:34197
--webserver_port=41727
--webserver_interface=127.6.2.194
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:11:41.939811 22312 raft_consensus.cc:348] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } } peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } }
I20260924 10:11:41.941236 22312 raft_consensus.cc:729] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbf040cea454ed8bcffb944fcc007f0, State: Initialized, Role: FOLLOWER
I20260924 10:11:41.942116 22312 consensus_queue.cc:260] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } } peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } }
I20260924 10:11:41.958531 22312 ts_tablet_manager.cc:1428] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Time spent starting tablet: real 0.030s user 0.014s sys 0.016s
Sep 24 10:11:41 dist-test-slave-s3n3 krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:11:41.976115 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:41.941572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:39555 (local address 127.6.2.254:34197)
0924 10:11:41.954980 (+ 13408us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:41.955003 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:41.955092 (+ 89us) server_negotiation.cc:407] Connection header received
0924 10:11:41.955591 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:41.955615 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:41.956191 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:41.956687 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:41.958965 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:41.960279 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:41.962838 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:41.963544 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:41.963681 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:41.968031 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:41.968039 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:41.968054 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:41.968098 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:41.971136 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:41.971982 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:41.971991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:41.971999 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:41.972171 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:41.972948 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:41.972956 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:41.972959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:41.973436 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:41.973631 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:41.974442 (+ 811us) server_negotiation.cc:299] Negotiation successful
0924 10:11:41.974819 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13065,"thread_start_us":12917,"threads_started":1}
I20260924 10:11:41.976132 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:41.941520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:39555)
0924 10:11:41.952029 (+ 10509us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:41.952104 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:41.952959 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:41.956809 (+ 3850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:41.956823 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:41.957823 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:11:41.958708 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:41.958743 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:41.960467 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:41.960475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:41.962672 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:41.962692 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:41.963751 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:41.963760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:41.964077 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:41.964842 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:11:41.964877 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:11:41.967710 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:11:41.971314 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:41.971326 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:41.971344 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:41.971833 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:41.972648 (+ 815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:41.972651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:41.972654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:41.972826 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:41.973587 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:11:41.973595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:11:41.973950 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 10:11:41.974257 (+ 307us) client_negotiation.cc:240] Negotiation successful
0924 10:11:41.974801 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10033,"thread_start_us":11808,"threads_started":1}
I20260924 10:11:41.979154 22420 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34197
I20260924 10:11:41.979542 22420 heartbeater.cc:464] Registering TS with master...
I20260924 10:11:41.980388 22420 heartbeater.cc:511] Master 127.6.2.254:34197 requested a full tablet report, sending...
I20260924 10:11:41.983721 22233 ts_manager.cc:194] Registered new tserver with Master: ddbf040cea454ed8bcffb944fcc007f0 (127.6.2.193:42707)
I20260924 10:11:41.988370 22233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:39555
I20260924 10:11:41.993341 22420 heartbeater.cc:503] Master 127.6.2.254:34197 was elected leader, sending a full tablet report...
Sep 24 10:11:41 dist-test-slave-s3n3 krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:11:42.006714 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:41.987026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:44362)
0924 10:11:41.990110 (+ 3084us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:41.990152 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:41.990332 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:41.991767 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:41.991777 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:41.992498 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:11:41.992757 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:41.992771 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:41.994287 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:41.994293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:41.995993 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:41.996009 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:41.996834 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:41.996841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:41.997038 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:41.997826 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:11:41.997850 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:11:42.000541 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:11:42.003901 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:42.003916 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:42.003923 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:42.004362 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:42.004774 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:42.004779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:42.004783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:42.004930 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:42.005543 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:11:42.005551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:11:42.005751 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 10:11:42.005932 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 10:11:42.006313 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2890,"thread_start_us":2759,"threads_started":1}
I20260924 10:11:42.006752 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:41.987354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.6.2.193:42707)
0924 10:11:41.990986 (+ 3632us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:41.990997 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:41.991037 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:11:41.991173 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:41.991179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:41.991482 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:41.991719 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:41.992930 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:41.994147 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:41.996147 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:41.996671 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:41.996790 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:42.000813 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:42.000825 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:42.000837 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:42.000869 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:42.003751 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:42.004525 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:42.004532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:42.004534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:42.004650 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:42.005051 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:42.005060 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:42.005062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:42.005402 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:42.005598 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:42.006095 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 10:11:42.006335 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3465,"thread_start_us":3268,"threads_started":1}
I20260924 10:11:42.008350 22355 tablet_service.cc:1514] Processing DeleteTablet for tablet dd5233a9e6464965850929afcd081086 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:11:20 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:44362
I20260924 10:11:42.010426 22428 tablet_replica.cc:330] stopping tablet replica
I20260924 10:11:42.010802 22428 raft_consensus.cc:2229] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:11:42.011240 22428 raft_consensus.cc:2258] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:11:42.012588 22428 ts_tablet_manager.cc:1905] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:11:42.019826 22428 ts_tablet_manager.cc:1918] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:11:42.019994 22428 log.cc:1196] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/dd5233a9e6464965850929afcd081086
I20260924 10:11:42.020458 22428 ts_tablet_manager.cc:1939] T dd5233a9e6464965850929afcd081086 P ddbf040cea454ed8bcffb944fcc007f0: Deleting consensus metadata
I20260924 10:11:42.022763 22217 catalog_manager.cc:4841] TS ddbf040cea454ed8bcffb944fcc007f0 (127.6.2.193:42707): tablet dd5233a9e6464965850929afcd081086 (table test-table [id=f98e912b39a3468e833cd6213119ddb0]) successfully deleted
W20260924 10:11:42.291538 22425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:11:42.291893 22425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:11:42.292058 22425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:11:42.299482 22425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:11:42.299643 22425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:11:42.299841 22425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:11:42.309880 22425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:40735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=41727
--tserver_master_addrs=127.6.2.254:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:11:42.311345 22425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:11:42.312723 22425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:11:42.326007 22433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:42.325996 22434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:42.326045 22436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:11:42.327044 22425 server_base.cc:1034] running on GCE node
I20260924 10:11:42.327664 22425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:11:42.328614 22425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:11:42.329821 22425 hybrid_clock.cc:648] HybridClock initialized: now 1790244702329767 us; error 45 us; skew 500 ppm
I20260924 10:11:42.330153 22425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:11:42 dist-test-slave-s3n3 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244702, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:11:42.333374 22425 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:11:42.335207 22425 webserver.cc:460] Webserver started at http://127.6.2.194:41727/ using document root <none> and password file <none>
I20260924 10:11:42.335938 22425 fs_manager.cc:359] Metadata directory not provided
I20260924 10:11:42.336051 22425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:11:42.341667 22425 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:11:42.344441 22443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:42.345842 22425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:11:42.346035 22425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "103da8104d9f496fb4998c589a5db906"
format_stamp: "Formatted at 2026-09-24 10:11:19 on dist-test-slave-s3n3"
server_key: "8806cf784631be0068056fc76852f253"
server_key_iv: "d9ab185082fa452d1da1492f31f53b3d"
server_key_version: "encryptionkey@0"
I20260924 10:11:42.347105 22425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:11:42.372027 22425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:11:42.376281 22425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:11:42.376585 22425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:11:42.377596 22425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:11:42.379024 22425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:11:42.379199 22425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:42.379333 22425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:11:42.379422 22425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:42.415390 22425 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:40735
I20260924 10:11:42.415431 22555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:40735 every 8 connection(s)
I20260924 10:11:42.417189 22425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:11:42.426496 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 22425
I20260924 10:11:42.427304 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:43631
--local_ip_for_outbound_sockets=127.6.2.195
--tserver_master_addrs=127.6.2.254:34197
--webserver_port=44749
--webserver_interface=127.6.2.195
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:11:42 dist-test-slave-s3n3 krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244702, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:11:42.438894 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:42.419876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:60247 (local address 127.6.2.254:34197)
0924 10:11:42.420083 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:42.420089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:42.421215 (+ 1126us) server_negotiation.cc:407] Connection header received
0924 10:11:42.423044 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:42.423050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:42.423171 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:42.423286 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:42.424834 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.425701 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.428031 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.428462 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.428559 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:42.433099 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:42.433105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:42.433108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:42.433142 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:42.435844 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:42.436576 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:42.436583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:42.436586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:42.436720 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:42.437151 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:42.437157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:42.437161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:42.437435 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:42.437588 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:42.438283 (+ 695us) server_negotiation.cc:299] Negotiation successful
0924 10:11:42.438529 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:11:42.439929 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:42.420206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:60247)
0924 10:11:42.421024 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:42.421075 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:42.422788 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:42.423535 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:42.423543 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:42.424100 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:11:42.424655 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:42.424672 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.425860 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.425864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:42.427884 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:42.427904 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.428700 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.428710 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:42.429038 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:42.429804 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:11:42.429838 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:11:42.432846 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:11:42.436000 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:42.436009 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:42.436027 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:42.436450 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:42.436847 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:42.436852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:42.436856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:42.437040 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:42.437561 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:11:42.437570 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:11:42.437938 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 10:11:42.438209 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 10:11:42.438740 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":346,"threads_started":1}
I20260924 10:11:42.441509 22556 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34197
I20260924 10:11:42.441864 22556 heartbeater.cc:464] Registering TS with master...
I20260924 10:11:42.442725 22556 heartbeater.cc:511] Master 127.6.2.254:34197 requested a full tablet report, sending...
I20260924 10:11:42.444439 22233 ts_manager.cc:194] Registered new tserver with Master: 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735)
I20260924 10:11:42.445760 22233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:60247
W20260924 10:11:42.762416 22559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:11:42.762717 22559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:11:42.762810 22559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:11:42.770181 22559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:11:42.770311 22559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:11:42.770428 22559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:11:42.780606 22559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:43631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=44749
--tserver_master_addrs=127.6.2.254:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:11:42.781875 22559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:11:42.783172 22559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:11:42.797310 22567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:42.797353 22564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:11:42.797267 22565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:11:42.798164 22559 server_base.cc:1034] running on GCE node
I20260924 10:11:42.798867 22559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:11:42.800052 22559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:11:42.801292 22559 hybrid_clock.cc:648] HybridClock initialized: now 1790244702801161 us; error 127 us; skew 500 ppm
I20260924 10:11:42.801630 22559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:11:42 dist-test-slave-s3n3 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244702, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:11:42.804965 22559 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:11:42.806707 22559 webserver.cc:460] Webserver started at http://127.6.2.195:44749/ using document root <none> and password file <none>
I20260924 10:11:42.808068 22559 fs_manager.cc:359] Metadata directory not provided
I20260924 10:11:42.808241 22559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:11:42.813984 22559 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:11:42.817119 22574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:42.818553 22559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:11:42.818737 22559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "216bcbd68e2c489991d5cc631c7c213c"
format_stamp: "Formatted at 2026-09-24 10:11:19 on dist-test-slave-s3n3"
server_key: "33759a5f759b6063ce32b8896eebddbf"
server_key_iv: "7a4e16f0c7c170897a668925480bd5b5"
server_key_version: "encryptionkey@0"
I20260924 10:11:42.819845 22559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:11:42.841806 22559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:11:42.846405 22559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:11:42.846715 22559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:11:42.847936 22559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:11:42.849383 22559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:11:42.849470 22559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:42.849634 22559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:11:42.849699 22559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:11:42.885703 22559 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:43631
I20260924 10:11:42.885762 22686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:43631 every 8 connection(s)
I20260924 10:11:42.887246 22559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:11:42.895919 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 22559
Sep 24 10:11:42 dist-test-slave-s3n3 krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244702, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:11:42.908981 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:42.889924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:43721 (local address 127.6.2.254:34197)
0924 10:11:42.890134 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:42.890141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:42.891403 (+ 1262us) server_negotiation.cc:407] Connection header received
0924 10:11:42.893230 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:42.893234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:42.893305 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:42.893401 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:42.894952 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.895997 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.898497 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.899124 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.899274 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:42.902783 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:42.902790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:42.902794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:42.902837 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:42.905877 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:42.906599 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:42.906606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:42.906612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:42.906734 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:42.907217 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:42.907222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:42.907226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:42.907532 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:42.907696 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:42.908403 (+ 707us) server_negotiation.cc:299] Negotiation successful
0924 10:11:42.908622 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:11:42.910035 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:42.890460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:43721)
0924 10:11:42.891246 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:42.891291 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:42.892955 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:42.893648 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:42.893657 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:42.894249 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:11:42.894783 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:42.894799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.896222 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.896228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:42.898335 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:42.898352 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.899374 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.899384 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:42.899640 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:42.900350 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:11:42.900377 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:11:42.902546 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:11:42.906015 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:42.906024 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:42.906042 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:42.906476 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:42.906856 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:11:42.906861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:11:42.906864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:11:42.907052 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:11:42.907676 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:11:42.907684 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:11:42.908042 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 10:11:42.908317 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 10:11:42.908868 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":305,"threads_started":1}
I20260924 10:11:42.911849 22687 heartbeater.cc:347] Connected to a master server at 127.6.2.254:34197
I20260924 10:11:42.912200 22687 heartbeater.cc:464] Registering TS with master...
I20260924 10:11:42.913118 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:42.898754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.6.2.254:34197)
0924 10:11:42.899285 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:42.899289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:42.899311 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:11:42.899350 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:42.899353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:42.899418 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:42.899543 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:42.901241 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.902083 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.903805 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.904263 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.904404 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:42.907118 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:42.907125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:42.907130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:42.907170 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:42.910028 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:42.910990 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:42.910998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:42.911003 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:42.911187 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:42.911652 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:42.911655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:42.911657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:42.912036 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:42.912224 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:42.912633 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 10:11:42.912841 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":297,"threads_started":1}
I20260924 10:11:42.913129 22687 heartbeater.cc:511] Master 127.6.2.254:34197 requested a full tablet report, sending...
I20260924 10:11:42.914702 22233 ts_manager.cc:194] Registered new tserver with Master: 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631)
I20260924 10:11:42.915755 22233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:43721
I20260924 10:11:42.916859 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:11:42.923575 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:42.917887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.6.2.254:34197)
0924 10:11:42.918114 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:42.918123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:42.918601 (+ 478us) server_negotiation.cc:407] Connection header received
0924 10:11:42.918820 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:42.918824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:42.918942 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:42.919143 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:11:42.919543 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.920226 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.921632 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.922054 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.922148 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:42.922685 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:11:42.922918 (+ 233us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 10:11:42.923328 (+ 410us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50878: token expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 10:11:42.923709 22691 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50878: token expired
W20260924 10:11:42.924451 22154 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.2.254:34197)
I20260924 10:11:42.938680 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:42.924918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.6.2.254:34197)
0924 10:11:42.925087 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:42.925092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:42.925106 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:11:42.925235 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:42.925238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:42.925293 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:42.925374 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:42.926652 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.927436 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.928703 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.929085 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.929173 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:42.931869 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:42.931874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:42.931878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:42.931910 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:42.934593 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:42.936045 (+ 1452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:42.936052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:42.936055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:42.936226 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:42.936907 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:42.936912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:42.936916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:42.937327 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:42.937591 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:42.938024 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 10:11:42.938285 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:11:42.942580 22232 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:11:42.950183 22232 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:11:42.963438 22355 tablet_service.cc:1467] Processing CreateTablet for tablet 5d916d70fbcd4d32be16b97d08e3bb48 (DEFAULT_TABLE table=test-table [id=bf4f1781751f4989a0968a1f6cd36afc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:11:42.964587 22355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d916d70fbcd4d32be16b97d08e3bb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:11:42.977854 22701 tablet_bootstrap.cc:492] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: Bootstrap starting.
I20260924 10:11:42.978081 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:42.961576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:40735 (local address 127.0.0.1:39454)
0924 10:11:42.964068 (+ 2492us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:42.964100 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:42.964264 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:42.965469 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:42.965478 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:42.965514 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:11:42.967354 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:42.967372 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.971365 (+ 3993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.971374 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:42.975715 (+ 4341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:42.975732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.977123 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.977129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:42.977320 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:42.977397 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 10:11:42.977494 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 10:11:42.977677 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2332,"thread_start_us":15959,"threads_started":1}
I20260924 10:11:42.978819 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:42.961663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39454 (local address 127.6.2.194:40735)
0924 10:11:42.962485 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:42.962495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:42.964428 (+ 1933us) server_negotiation.cc:407] Connection header received
0924 10:11:42.964571 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:42.964582 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:42.964930 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:42.965166 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:11:42.967639 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.971150 (+ 3511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.975896 (+ 4746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.976876 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.977053 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:42.977293 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:42.978302 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0924 10:11:42.978512 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":526,"threads_started":1}
I20260924 10:11:42.980072 22701 tablet_bootstrap.cc:654] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: Neither blocks nor log segments found. Creating new log.
I20260924 10:11:42.982811 22701 tablet_bootstrap.cc:492] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: No bootstrap required, opened a new log
I20260924 10:11:42.982969 22701 ts_tablet_manager.cc:1397] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260924 10:11:42.983336 22491 tablet_service.cc:1467] Processing CreateTablet for tablet 5d916d70fbcd4d32be16b97d08e3bb48 (DEFAULT_TABLE table=test-table [id=bf4f1781751f4989a0968a1f6cd36afc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:11:42.983875 22701 raft_consensus.cc:348] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:11:42.984179 22701 raft_consensus.cc:374] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:11:42.984304 22701 raft_consensus.cc:729] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbf040cea454ed8bcffb944fcc007f0, State: Initialized, Role: FOLLOWER
I20260924 10:11:42.984601 22701 consensus_queue.cc:260] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:11:42.984911 22491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d916d70fbcd4d32be16b97d08e3bb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:11:42.985579 22701 ts_tablet_manager.cc:1428] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 10:11:42.986555 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:42.962650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.6.2.195:43631)
0924 10:11:42.975568 (+ 12918us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:42.975578 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:42.977857 (+ 2279us) server_negotiation.cc:407] Connection header received
0924 10:11:42.977991 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:42.978003 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:42.978337 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:42.978545 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:11:42.979560 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.980980 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.983276 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.984531 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.984688 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:42.984948 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:42.985969 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0924 10:11:42.986165 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12727,"thread_start_us":12935,"threads_started":1}
I20260924 10:11:42.987200 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:42.962029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:43631 (local address 127.0.0.1:44808)
0924 10:11:42.977198 (+ 15169us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:42.977233 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:42.977432 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:42.978835 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:42.978838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:42.978861 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:11:42.979330 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:42.979344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.981134 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.981142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:42.983097 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:42.983114 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:42.984766 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:42.984773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:42.984996 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:42.985041 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 10:11:42.986557 (+ 1516us) client_negotiation.cc:240] Negotiation successful
0924 10:11:42.986724 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14957,"thread_start_us":1716,"threads_started":1}
I20260924 10:11:42.991412 22622 tablet_service.cc:1467] Processing CreateTablet for tablet 5d916d70fbcd4d32be16b97d08e3bb48 (DEFAULT_TABLE table=test-table [id=bf4f1781751f4989a0968a1f6cd36afc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:11:42.992993 22622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d916d70fbcd4d32be16b97d08e3bb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:11:43.001139 22704 tablet_bootstrap.cc:492] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906: Bootstrap starting.
I20260924 10:11:43.005209 22704 tablet_bootstrap.cc:654] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906: Neither blocks nor log segments found. Creating new log.
I20260924 10:11:43.006328 22704 log.cc:824] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906: Log is configured to *not* fsync() on all Append() calls
I20260924 10:11:43.007153 22705 tablet_bootstrap.cc:492] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c: Bootstrap starting.
I20260924 10:11:43.009444 22704 tablet_bootstrap.cc:492] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906: No bootstrap required, opened a new log
I20260924 10:11:43.009737 22704 ts_tablet_manager.cc:1397] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 10:11:43.010113 22705 tablet_bootstrap.cc:654] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c: Neither blocks nor log segments found. Creating new log.
I20260924 10:11:43.011209 22705 log.cc:824] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:11:43.014176 22705 tablet_bootstrap.cc:492] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c: No bootstrap required, opened a new log
I20260924 10:11:43.014443 22705 ts_tablet_manager.cc:1397] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260924 10:11:43.018466 22704 raft_consensus.cc:348] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:11:43.019054 22704 raft_consensus.cc:374] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:11:43.019209 22704 raft_consensus.cc:729] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103da8104d9f496fb4998c589a5db906, State: Initialized, Role: FOLLOWER
I20260924 10:11:43.019866 22704 consensus_queue.cc:260] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:11:43.021018 22556 heartbeater.cc:503] Master 127.6.2.254:34197 was elected leader, sending a full tablet report...
I20260924 10:11:43.021564 22704 ts_tablet_manager.cc:1428] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 10:11:43.021982 22705 raft_consensus.cc:348] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:11:43.022326 22705 raft_consensus.cc:374] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:11:43.022450 22705 raft_consensus.cc:729] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216bcbd68e2c489991d5cc631c7c213c, State: Initialized, Role: FOLLOWER
I20260924 10:11:43.023141 22705 consensus_queue.cc:260] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:11:43.024207 22687 heartbeater.cc:503] Master 127.6.2.254:34197 was elected leader, sending a full tablet report...
I20260924 10:11:43.025921 22705 ts_tablet_manager.cc:1428] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
W20260924 10:11:43.139009 22688 tablet.cc:2369] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:11:43.169304 22557 tablet.cc:2369] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:11:43.179818 22703 raft_consensus.cc:482] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:11:43.180200 22703 raft_consensus.cc:504] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:11:43.181885 22703 leader_election.cc:288] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735), 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631)
I20260924 10:11:43.188351 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.182432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:41401 (local address 127.6.2.194:40735)
0924 10:11:43.182625 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:43.182631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:43.183103 (+ 472us) server_negotiation.cc:407] Connection header received
0924 10:11:43.183360 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:43.183364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:43.183426 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:43.183524 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:11:43.184109 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.184807 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.186578 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.187414 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.187566 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:43.187668 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:43.187951 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 10:11:43.188081 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:11:43.188439 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.182214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:40735 (local address 127.6.2.193:41401)
0924 10:11:43.182937 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:43.182967 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:43.183187 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:43.183553 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:43.183558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:43.183584 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:11:43.183928 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:43.183938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.184959 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.184964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:43.186413 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:43.186426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.187597 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.187603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:43.187768 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:43.187808 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 10:11:43.187924 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 10:11:43.188069 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":611,"threads_started":1}
I20260924 10:11:43.188535 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.182397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:40829 (local address 127.6.2.195:43631)
0924 10:11:43.182614 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:43.182621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:43.183273 (+ 652us) server_negotiation.cc:407] Connection header received
0924 10:11:43.183473 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:43.183477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:43.183535 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:43.183639 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:11:43.184314 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.185011 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.186763 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.187602 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.187760 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:43.187868 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:43.188157 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 10:11:43.188284 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:11:43.188918 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.182301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:43631 (local address 127.6.2.193:40829)
0924 10:11:43.183159 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:43.183179 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:11:43.183319 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:11:43.183658 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:11:43.183662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:11:43.183702 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:11:43.184152 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:43.184161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.185150 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.185157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:43.186582 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:11:43.186594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.187815 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.187821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:11:43.187989 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:43.188021 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:11:43.188127 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 10:11:43.188267 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":328,"threads_started":1}
I20260924 10:11:43.189785 22511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d916d70fbcd4d32be16b97d08e3bb48" candidate_uuid: "ddbf040cea454ed8bcffb944fcc007f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103da8104d9f496fb4998c589a5db906" is_pre_election: true
I20260924 10:11:43.189965 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d916d70fbcd4d32be16b97d08e3bb48" candidate_uuid: "ddbf040cea454ed8bcffb944fcc007f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216bcbd68e2c489991d5cc631c7c213c" is_pre_election: true
I20260924 10:11:43.190282 22511 raft_consensus.cc:2454] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbf040cea454ed8bcffb944fcc007f0 in term 0.
I20260924 10:11:43.190521 22642 raft_consensus.cc:2454] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbf040cea454ed8bcffb944fcc007f0 in term 0.
W20260924 10:11:43.191138 22421 tablet.cc:2369] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:11:43.191116 22307 leader_election.cc:302] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103da8104d9f496fb4998c589a5db906, ddbf040cea454ed8bcffb944fcc007f0; no voters:
I20260924 10:11:43.191601 22703 raft_consensus.cc:2781] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:11:43.191776 22703 raft_consensus.cc:482] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:11:43.191888 22703 raft_consensus.cc:3037] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:11:43.194054 22703 raft_consensus.cc:504] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:11:43.194782 22703 leader_election.cc:288] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 1 election: Requested vote from peers 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735), 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631)
I20260924 10:11:43.195291 22511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d916d70fbcd4d32be16b97d08e3bb48" candidate_uuid: "ddbf040cea454ed8bcffb944fcc007f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103da8104d9f496fb4998c589a5db906"
I20260924 10:11:43.195291 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d916d70fbcd4d32be16b97d08e3bb48" candidate_uuid: "ddbf040cea454ed8bcffb944fcc007f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216bcbd68e2c489991d5cc631c7c213c"
I20260924 10:11:43.195506 22511 raft_consensus.cc:3037] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:11:43.195506 22642 raft_consensus.cc:3037] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:11:43.198771 22511 raft_consensus.cc:2454] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbf040cea454ed8bcffb944fcc007f0 in term 1.
I20260924 10:11:43.198771 22642 raft_consensus.cc:2454] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbf040cea454ed8bcffb944fcc007f0 in term 1.
I20260924 10:11:43.199245 22309 leader_election.cc:302] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216bcbd68e2c489991d5cc631c7c213c, ddbf040cea454ed8bcffb944fcc007f0; no voters:
I20260924 10:11:43.199604 22703 raft_consensus.cc:2781] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:11:43.200156 22703 raft_consensus.cc:686] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 LEADER]: Becoming Leader. State: Replica: ddbf040cea454ed8bcffb944fcc007f0, State: Running, Role: LEADER
I20260924 10:11:43.200877 22703 consensus_queue.cc:237] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:11:43.205466 22232 catalog_manager.cc:5495] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddbf040cea454ed8bcffb944fcc007f0 (127.6.2.193). New cstate: current_term: 1 leader_uuid: "ddbf040cea454ed8bcffb944fcc007f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } health_report { overall_health: HEALTHY } } }
I20260924 10:11:43.270340 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.264159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.6.2.193:42707)
0924 10:11:43.264935 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:43.264940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:43.264960 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:11:43.265049 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:43.265053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:43.265129 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:43.265308 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:11:43.265751 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.266540 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.268018 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.268460 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.268597 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:43.269068 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:11:43.269659 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:11:43.269839 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:43.269898 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 10:11:43.269985 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":498,"threads_started":1}
I20260924 10:11:43.277062 22642 raft_consensus.cc:1261] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c [term 1 FOLLOWER]: Refusing update from remote peer ddbf040cea454ed8bcffb944fcc007f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:11:43.277062 22511 raft_consensus.cc:1261] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906 [term 1 FOLLOWER]: Refusing update from remote peer ddbf040cea454ed8bcffb944fcc007f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:11:43.278131 22712 consensus_queue.cc:1035] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:11:43.278548 22703 consensus_queue.cc:1035] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:11:43.292435 22717 mvcc.cc:205] Tried to move back new op lower bound from 7332842304612159488 to 7332842304314449920. Current Snapshot: MvccSnapshot[applied={T|T < 7332842304612159488}]
I20260924 10:11:43.293751 22716 mvcc.cc:205] Tried to move back new op lower bound from 7332842304612159488 to 7332842304314449920. Current Snapshot: MvccSnapshot[applied={T|T < 7332842304612159488}]
I20260924 10:11:43.309228 22232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:11:43.309528 22232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:11:43.313146 22232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Sending DeleteTablet for 3 replicas of tablet 5d916d70fbcd4d32be16b97d08e3bb48
I20260924 10:11:43.314172 22491 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d916d70fbcd4d32be16b97d08e3bb48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:11:43 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:39454
I20260924 10:11:43.314297 22622 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d916d70fbcd4d32be16b97d08e3bb48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:11:43 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:44808
I20260924 10:11:43.314863 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 22270
I20260924 10:11:43.314822 22355 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d916d70fbcd4d32be16b97d08e3bb48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:11:43 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:44362
I20260924 10:11:43.329599 22726 tablet_replica.cc:330] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c: stopping tablet replica
I20260924 10:11:43.330300 22726 raft_consensus.cc:2229] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:11:43.331140 22726 raft_consensus.cc:2258] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:11:43.332409 22727 tablet_replica.cc:330] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: stopping tablet replica
I20260924 10:11:43.332707 22727 raft_consensus.cc:2229] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:11:43.333491 22727 raft_consensus.cc:2258] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:11:43.333774 22726 ts_tablet_manager.cc:1905] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:11:43.334121 22725 tablet_replica.cc:330] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906: stopping tablet replica
I20260924 10:11:43.334578 22725 raft_consensus.cc:2229] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:11:43.335131 22725 raft_consensus.cc:2258] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:11:43.336694 22727 ts_tablet_manager.cc:1905] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:11:43.337736 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.315658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.6.2.193:42707)
0924 10:11:43.316335 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:43.316343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:43.316366 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:11:43.316434 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:43.316442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:43.316542 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:43.316666 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:43.318774 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.320172 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.323301 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.324271 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.324425 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:43.328852 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:43.328861 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:43.328869 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:43.328928 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:43.333453 (+ 4525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:43.334407 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:43.334412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:43.334416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:43.334569 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:43.335107 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:43.335112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:43.335116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:43.335463 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:43.336943 (+ 1480us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:43.337077 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 10:11:43.337309 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530}
I20260924 10:11:43.338626 22415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:11:43.341418 22725 ts_tablet_manager.cc:1905] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:11:43.342857 22726 ts_tablet_manager.cc:1918] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:11:43.343086 22726 log.cc:1196] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/5d916d70fbcd4d32be16b97d08e3bb48
I20260924 10:11:43.343639 22726 ts_tablet_manager.cc:1939] T 5d916d70fbcd4d32be16b97d08e3bb48 P 216bcbd68e2c489991d5cc631c7c213c: Deleting consensus metadata
I20260924 10:11:43.345536 22220 catalog_manager.cc:4841] TS 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631): tablet 5d916d70fbcd4d32be16b97d08e3bb48 (table test-table [id=bf4f1781751f4989a0968a1f6cd36afc]) successfully deleted
I20260924 10:11:43.348630 22725 ts_tablet_manager.cc:1918] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:11:43.348817 22725 log.cc:1196] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/5d916d70fbcd4d32be16b97d08e3bb48
I20260924 10:11:43.349334 22725 ts_tablet_manager.cc:1939] T 5d916d70fbcd4d32be16b97d08e3bb48 P 103da8104d9f496fb4998c589a5db906: Deleting consensus metadata
I20260924 10:11:43.351250 22218 catalog_manager.cc:4841] TS 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735): tablet 5d916d70fbcd4d32be16b97d08e3bb48 (table test-table [id=bf4f1781751f4989a0968a1f6cd36afc]) successfully deleted
I20260924 10:11:43.452142 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 22270
I20260924 10:11:43.472086 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 22425
W20260924 10:11:43.472031 22217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.2.193:42707 (error 108)
I20260924 10:11:43.473932 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.473438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:44390)
0924 10:11:43.473631 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:43.473742 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 10:11:43.474179 22217 catalog_manager.cc:4584] TS ddbf040cea454ed8bcffb944fcc007f0 (127.6.2.193:42707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d916d70fbcd4d32be16b97d08e3bb48: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
I20260924 10:11:43.496927 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.473316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39460 (local address 127.6.2.194:40735)
0924 10:11:43.473514 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:43.473521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:43.473548 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:11:43.473618 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:43.473624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:43.473734 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:43.473854 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:43.475342 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.476120 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.478062 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.478559 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.478667 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:43.491529 (+ 12862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:43.491547 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:43.491565 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:43.491608 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:43.494296 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:43.495001 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:43.495009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:43.495011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:43.495244 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:43.495587 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:43.495591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:43.495593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:43.496005 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:43.496228 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:43.496480 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 10:11:43.496647 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:11:43.497681 22551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:11:43.527637 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.527207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:44402)
0924 10:11:43.527409 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:43.527498 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 10:11:43.528594 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.528261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:44408)
0924 10:11:43.528407 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:43.528477 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:11:43.589285 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.588763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:44414)
0924 10:11:43.588964 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:43.589071 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 10:11:43.589726 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 22425
I20260924 10:11:43.590768 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.590358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:44418)
0924 10:11:43.590522 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:43.590611 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 10:11:43.607252 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 22559
I20260924 10:11:43.620957 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.608008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.6.2.195:43631)
0924 10:11:43.608203 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 10:11:43.608210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:11:43.608231 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:11:43.608377 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:11:43.608382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:11:43.608459 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:11:43.608565 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:11:43.609871 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.610583 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.612111 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:11:43.612516 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:11:43.612651 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:11:43.615181 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:11:43.615193 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:11:43.615210 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:11:43.615243 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:11:43.617989 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:43.618700 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:43.618716 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:43.618720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:43.618871 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:11:43.619401 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:11:43.619409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:11:43.619413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:11:43.619776 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:11:43.620071 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 10:11:43.620388 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 10:11:43.620605 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:11:43.621592 22682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:11:43.671942 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.671334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:44422)
0924 10:11:43.671601 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:43.671726 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 10:11:43.673048 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.672652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:44432)
0924 10:11:43.672855 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:43.672933 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 10:11:43.713357 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 22559
I20260924 10:11:43.730456 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 22200
I20260924 10:11:43.731635 22263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:11:43.847944 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:11:43.847358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42707 (local address 127.0.0.1:44438)
0924 10:11:43.847625 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 10:11:43.847761 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 10:11:43.848155 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 22200
I20260924 10:12:03.870523 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34197
--webserver_interface=127.6.2.254
--webserver_port=35085
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:34197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:04.214891 22733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:04.215322 22733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:04.215494 22733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:04.223584 22733 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:12:04.223690 22733 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:12:04.223767 22733 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:12:04.223850 22733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:04.223948 22733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:04.224069 22733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:04.224159 22733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:04.224239 22733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:04.234321 22733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:34197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:34197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=35085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:04.235718 22733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:04.237020 22733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:04.249392 22738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:04.249358 22741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:04.249426 22739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:04.250319 22733 server_base.cc:1034] running on GCE node
I20260924 10:12:04.251150 22733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:04.252599 22733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:04.253829 22733 hybrid_clock.cc:648] HybridClock initialized: now 1790244724253755 us; error 57 us; skew 500 ppm
I20260924 10:12:04.254172 22733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:04 dist-test-slave-s3n3 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:04.265000 22733 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:12:04.266628 22733 webserver.cc:460] Webserver started at http://127.6.2.254:35085/ using document root <none> and password file <none>
I20260924 10:12:04.267449 22733 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:04.267560 22733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:04.273068 22733 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:12:04.275880 22748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:04.277170 22733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:12:04.277338 22733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "b6205e8b4b0547cf9d87f7e883027d1e"
format_stamp: "Formatted at 2026-09-24 10:11:18 on dist-test-slave-s3n3"
server_key: "60cb54d30ad13703676abcb157b31492"
server_key_iv: "ead131043e2cbfecc08f3ab42b052d1b"
server_key_version: "encryptionkey@0"
I20260924 10:12:04.278358 22733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:04.308388 22733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:04.312920 22733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:04.313225 22733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:04.332218 22733 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:34197
I20260924 10:12:04.332265 22799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:34197 every 8 connection(s)
I20260924 10:12:04.333716 22733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:04.335469 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 22733
I20260924 10:12:04.336467 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:42707
--local_ip_for_outbound_sockets=127.6.2.193
--tserver_master_addrs=127.6.2.254:34197
--webserver_port=44267
--webserver_interface=127.6.2.193
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:12:04.343348 22800 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:12:04.347460 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Bootstrap starting.
I20260924 10:12:04.368242 22800 log.cc:824] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:04.384114 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:12:04.384668 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Bootstrap complete.
I20260924 10:12:04.393620 22800 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } }
I20260924 10:12:04.394516 22800 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6205e8b4b0547cf9d87f7e883027d1e, State: Initialized, Role: FOLLOWER
I20260924 10:12:04.395138 22800 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } }
I20260924 10:12:04.395433 22800 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:04.395582 22800 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:04.395768 22800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 2 FOLLOWER]: Advancing to term 3
I20260924 10:12:04.399457 22800 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } }
I20260924 10:12:04.399935 22800 leader_election.cc:302] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6205e8b4b0547cf9d87f7e883027d1e; no voters:
I20260924 10:12:04.400660 22800 leader_election.cc:288] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 10:12:04.400804 22804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 3 FOLLOWER]: Leader election won for term 3
I20260924 10:12:04.402022 22804 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [term 3 LEADER]: Becoming Leader. State: Replica: b6205e8b4b0547cf9d87f7e883027d1e, State: Running, Role: LEADER
I20260924 10:12:04.402608 22804 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } }
I20260924 10:12:04.402866 22800 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:04.405068 22806 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6205e8b4b0547cf9d87f7e883027d1e. Latest consensus state: current_term: 3 leader_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } } }
I20260924 10:12:04.405527 22806 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:04.406123 22805 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6205e8b4b0547cf9d87f7e883027d1e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 34197 } } }
I20260924 10:12:04.406394 22805 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:04.406437 22810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:04.418270 22810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:04.420286 22810 catalog_manager.cc:1259] Loaded cluster ID: d7095e5eb7fa45a89db8ec96da5674be
I20260924 10:12:04.420473 22810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:04.425668 22810 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:04.428000 22810 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Loaded TSK: 0
I20260924 10:12:04.428212 22810 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b6205e8b4b0547cf9d87f7e883027d1e: Loaded TSK: 1
F20260924 10:12:04.429606 22810 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2eb645e9ad google::LogMessage::Fail() at ??:0
@ 0x7f2eb646188a google::LogMessage::SendToLog() at ??:0
@ 0x7f2eb645e4de google::LogMessage::Flush() at ??:0
@ 0x7f2eb645fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2eba1c27d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2eba1b173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2ecbe911c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2ecbe90783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2ecbf1c2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f2ecbf1c0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f2ecbf665e9 std::function<>::operator()() at ??:0
@ 0x7f2ecbe94cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f2ecbe92c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2ecbf1a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f2ecbf1a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2eb90aa345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2eb90ad91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2eb90ad799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2eb9088eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2ebc3626db start_thread at ??:0
@ 0x7f2eb52f861f clone at ??:0
W20260924 10:12:04.755615 22802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:04.755899 22802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:04.755995 22802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:04.763489 22802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:04.763602 22802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:04.763715 22802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:12:04.773636 22802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:42707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=44267
--tserver_master_addrs=127.6.2.254:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:04.774875 22802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:04.776193 22802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:04.789145 22848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:04.789244 22851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:04.789283 22849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:04.789922 22802 server_base.cc:1034] running on GCE node
I20260924 10:12:04.790568 22802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:04.791606 22802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:04.792847 22802 hybrid_clock.cc:648] HybridClock initialized: now 1790244724792792 us; error 75 us; skew 500 ppm
I20260924 10:12:04.793181 22802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:04 dist-test-slave-s3n3 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:04.796343 22802 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:12:04.797894 22802 webserver.cc:460] Webserver started at http://127.6.2.193:44267/ using document root <none> and password file <none>
I20260924 10:12:04.798575 22802 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:04.798718 22802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:04.804162 22802 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 10:12:04.807273 22858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:04.808645 22802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:12:04.808920 22802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "ddbf040cea454ed8bcffb944fcc007f0"
format_stamp: "Formatted at 2026-09-24 10:11:18 on dist-test-slave-s3n3"
server_key: "be9d8fe23b276fb565f7e98aa8a637b8"
server_key_iv: "35a7cd5bb477cbf9d05f798096c3a7d5"
server_key_version: "encryptionkey@0"
I20260924 10:12:04.809890 22802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:04.827595 22802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:04.832005 22802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:04.832294 22802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:04.833324 22802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:04.835244 22865 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 10:12:04.840993 22802 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:12:04.841157 22802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260924 10:12:04.841307 22802 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:12:04.844565 22802 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:12:04.844712 22802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260924 10:12:04.845116 22865 tablet_bootstrap.cc:492] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: Bootstrap starting.
I20260924 10:12:04.853734 22865 log.cc:824] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:04.869666 22865 tablet_bootstrap.cc:492] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:12:04.870647 22865 tablet_bootstrap.cc:492] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: Bootstrap complete.
I20260924 10:12:04.871304 22865 ts_tablet_manager.cc:1397] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.012s
I20260924 10:12:04.882346 22865 raft_consensus.cc:348] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:12:04.883728 22865 raft_consensus.cc:729] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbf040cea454ed8bcffb944fcc007f0, State: Initialized, Role: FOLLOWER
I20260924 10:12:04.884325 22802 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:42707
I20260924 10:12:04.884370 22972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:42707 every 8 connection(s)
I20260924 10:12:04.884613 22865 consensus_queue.cc:260] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:12:04.885726 22802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:12:04.886334 22865 ts_tablet_manager.cc:1428] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260924 10:12:04.890589 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:04.888853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:55477)
0924 10:12:04.889622 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:04.889807 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":349,"threads_started":1}
W20260924 10:12:04.891383 22974 heartbeater.cc:644] Failed to heartbeat to 127.6.2.254:34197 (0 consecutive failures): Network error: Failed to ping master at 127.6.2.254:34197: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
I20260924 10:12:04.892210 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:04.891873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:39511)
0924 10:12:04.892028 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:04.892096 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 10:12:04.893077 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:04.892703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:37395)
0924 10:12:04.892876 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:04.892956 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 10:12:04.893314 22974 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:12:04.896296 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 22802
I20260924 10:12:04.896908 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:40735
--local_ip_for_outbound_sockets=127.6.2.194
--tserver_master_addrs=127.6.2.254:34197
--webserver_port=41727
--webserver_interface=127.6.2.194
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:05.226907 22977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:05.227209 22977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:05.227397 22977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:05.234726 22977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:05.234830 22977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:05.235030 22977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:12:05.244771 22977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:40735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=41727
--tserver_master_addrs=127.6.2.254:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:05.246070 22977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:05.247359 22977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:05.260900 22982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:05.260946 22983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:05.260918 22985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:05.261780 22977 server_base.cc:1034] running on GCE node
I20260924 10:12:05.262432 22977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:05.263484 22977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:05.264775 22977 hybrid_clock.cc:648] HybridClock initialized: now 1790244725264682 us; error 51 us; skew 500 ppm
I20260924 10:12:05.265100 22977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:05 dist-test-slave-s3n3 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244725, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:05.268290 22977 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:12:05.269784 22977 webserver.cc:460] Webserver started at http://127.6.2.194:41727/ using document root <none> and password file <none>
I20260924 10:12:05.270484 22977 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:05.270634 22977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:05.276181 22977 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:12:05.278898 22992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:05.280478 22977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:12:05.280650 22977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "103da8104d9f496fb4998c589a5db906"
format_stamp: "Formatted at 2026-09-24 10:11:19 on dist-test-slave-s3n3"
server_key: "8806cf784631be0068056fc76852f253"
server_key_iv: "d9ab185082fa452d1da1492f31f53b3d"
server_key_version: "encryptionkey@0"
I20260924 10:12:05.281636 22977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:05.296267 22977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:05.300608 22977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:05.300904 22977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:05.301859 22977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:05.303252 22977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:05.303339 22977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:05.303491 22977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:05.303592 22977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:05.336645 22977 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:40735
I20260924 10:12:05.336696 23104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:40735 every 8 connection(s)
I20260924 10:12:05.338073 22977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:12:05.340893 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 22977
I20260924 10:12:05.341585 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:43631
--local_ip_for_outbound_sockets=127.6.2.195
--tserver_master_addrs=127.6.2.254:34197
--webserver_port=44749
--webserver_interface=127.6.2.195
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:12:05.343375 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:05.341103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:35549)
0924 10:12:05.341961 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:05.342202 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":302,"threads_started":1}
W20260924 10:12:05.344094 23105 heartbeater.cc:644] Failed to heartbeat to 127.6.2.254:34197 (0 consecutive failures): Network error: Failed to ping master at 127.6.2.254:34197: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
I20260924 10:12:05.345013 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:05.344596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:44757)
0924 10:12:05.344778 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:05.344869 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 10:12:05.345976 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:05.345556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:34031)
0924 10:12:05.345722 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:05.345819 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 10:12:05.346232 23105 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:12:05.661733 23108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:05.662021 23108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:05.662187 23108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:05.670054 23108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:05.670169 23108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:05.670342 23108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:12:05.680330 23108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:43631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=44749
--tserver_master_addrs=127.6.2.254:34197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:05.681630 23108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:05.682910 23108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:05.696740 23116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:05.696833 23113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:05.696861 23114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:05.697503 23108 server_base.cc:1034] running on GCE node
I20260924 10:12:05.698202 23108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:05.699288 23108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:05.700562 23108 hybrid_clock.cc:648] HybridClock initialized: now 1790244725700377 us; error 176 us; skew 500 ppm
I20260924 10:12:05.700958 23108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:05 dist-test-slave-s3n3 krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244725, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:05.704239 23108 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:12:05.705956 23108 webserver.cc:460] Webserver started at http://127.6.2.195:44749/ using document root <none> and password file <none>
I20260924 10:12:05.706621 23108 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:05.706720 23108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:05.712029 23108 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:12:05.714977 23123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:05.716327 23108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 10:12:05.716459 23108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "216bcbd68e2c489991d5cc631c7c213c"
format_stamp: "Formatted at 2026-09-24 10:11:19 on dist-test-slave-s3n3"
server_key: "33759a5f759b6063ce32b8896eebddbf"
server_key_iv: "7a4e16f0c7c170897a668925480bd5b5"
server_key_version: "encryptionkey@0"
I20260924 10:12:05.717406 23108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:05.737416 23108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:05.741861 23108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:05.742086 23108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:05.743002 23108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:05.744325 23108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:05.744413 23108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:05.744488 23108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:05.744542 23108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:05.779212 23108 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:43631
I20260924 10:12:05.779263 23235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:43631 every 8 connection(s)
I20260924 10:12:05.780607 23108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:12:05.785707 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:05.783764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:40135)
0924 10:12:05.784507 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:05.784699 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":289,"threads_started":1}
I20260924 10:12:05.786494 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 23108
W20260924 10:12:05.786561 23236 heartbeater.cc:644] Failed to heartbeat to 127.6.2.254:34197 (0 consecutive failures): Network error: Failed to ping master at 127.6.2.254:34197: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
I20260924 10:12:05.787639 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:05.787213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:44819)
0924 10:12:05.787396 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:05.787481 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 10:12:05.788656 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:05.788318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:42451)
0924 10:12:05.788476 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:05.788549 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 10:12:05.788921 23236 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:12:05.894520 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:05.893803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:53291)
0924 10:12:05.894227 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:05.894310 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":195,"threads_started":1}
I20260924 10:12:06.320890 23241 raft_consensus.cc:482] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:12:06.321189 23241 raft_consensus.cc:504] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:12:06.322819 23241 leader_election.cc:288] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735), 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631)
Sep 24 10:12:06 dist-test-slave-s3n3 krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
Sep 24 10:12:06 dist-test-slave-s3n3 krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:12:06.346449 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:06.323266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:43631 (local address 127.6.2.193:57461)
0924 10:12:06.323647 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:06.323704 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:06.324397 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:06.328335 (+ 3938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:06.328348 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:06.329259 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:06.329956 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:06.329978 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:06.331635 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:06.331638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:06.333065 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:06.333078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:06.334106 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:06.334113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:06.334354 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:06.335321 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:12:06.335357 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:06.338888 (+ 3531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:06.343253 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:06.343259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:06.343266 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:06.343634 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:06.344096 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:06.344101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:06.344104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:06.344242 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:06.344952 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:06.344957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:06.345142 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 10:12:06.345285 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 10:12:06.345796 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":193,"spinlock_wait_cycles":19328}
I20260924 10:12:06.346519 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:06.323500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:57461 (local address 127.6.2.195:43631)
0924 10:12:06.325832 (+ 2332us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:06.325844 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:06.325891 (+ 47us) server_negotiation.cc:407] Connection header received
0924 10:12:06.327452 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:06.327462 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:06.327831 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:06.328103 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:06.330189 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:06.331424 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:06.333216 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:06.333847 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:06.334083 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:06.339247 (+ 5164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:06.339258 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:06.339274 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:06.339327 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:06.343112 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:06.343752 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:06.343762 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:06.343772 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:06.343957 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:06.344352 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:06.344359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:06.344362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:06.344802 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:06.344992 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:06.345525 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 10:12:06.346034 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2122,"thread_start_us":1982,"threads_started":1}
I20260924 10:12:06.346457 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:06.323306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:40735 (local address 127.6.2.193:35203)
0924 10:12:06.326087 (+ 2781us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:06.326107 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:06.326284 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:06.328741 (+ 2457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:06.328745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:06.329258 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:06.329987 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:06.330000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:06.331621 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:06.331625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:06.333067 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:06.333080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:06.334054 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:06.334061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:06.334353 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:06.335430 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:12:06.335447 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:06.338887 (+ 3440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:06.342658 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:06.342668 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:06.342682 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:06.343200 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:06.343682 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:06.343690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:06.343694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:06.343901 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:06.344683 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:06.344691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:06.344985 (+ 294us) client_negotiation.cc:769] Sending connection context
0924 10:12:06.345245 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 10:12:06.346006 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2644,"mutex_wait_us":382,"spinlock_wait_cycles":19584,"thread_start_us":2473,"threads_started":1}
I20260924 10:12:06.347558 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:06.323465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:35203 (local address 127.6.2.194:40735)
0924 10:12:06.325304 (+ 1839us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:06.325323 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:06.326227 (+ 904us) server_negotiation.cc:407] Connection header received
0924 10:12:06.327970 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:06.327978 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:06.328404 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:06.328745 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:06.330194 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:06.331431 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:06.333223 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:06.333849 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:06.334020 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:06.339235 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:06.339244 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:06.339258 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:06.339295 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:06.342491 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:06.343324 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:06.343334 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:06.343342 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:06.343547 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:06.344032 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:06.344036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:06.344038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:06.344531 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:06.344720 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:06.345473 (+ 753us) server_negotiation.cc:299] Negotiation successful
0924 10:12:06.345828 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1601,"thread_start_us":1447,"threads_started":1}
I20260924 10:12:06.348296 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:06.346704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:33219)
0924 10:12:06.347743 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:06.347929 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":827,"thread_start_us":681,"threads_started":1}
W20260924 10:12:06.350400 22862 leader_election.cc:341] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631): Not found: Tablet not found: 5d916d70fbcd4d32be16b97d08e3bb48
W20260924 10:12:06.350669 22860 leader_election.cc:341] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735): Not found: Tablet not found: 5d916d70fbcd4d32be16b97d08e3bb48
I20260924 10:12:06.350898 22860 leader_election.cc:302] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddbf040cea454ed8bcffb944fcc007f0; no voters: 103da8104d9f496fb4998c589a5db906, 216bcbd68e2c489991d5cc631c7c213c
I20260924 10:12:06.351362 23241 raft_consensus.cc:2726] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:12:06.790321 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:06.789429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:43133)
0924 10:12:06.789929 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:06.790064 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":245,"threads_started":1}
I20260924 10:12:06.896371 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:06.895601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:40095)
0924 10:12:06.896084 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:06.896168 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":250,"threads_started":1}
I20260924 10:12:07.350102 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:07.349435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:50937)
0924 10:12:07.349865 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:07.349946 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":234,"threads_started":1}
I20260924 10:12:07.792076 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:07.791371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:41899)
0924 10:12:07.791831 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:07.791915 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":223,"threads_started":1}
I20260924 10:12:07.898126 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:07.897349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:49921)
0924 10:12:07.897861 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:07.897945 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":226,"threads_started":1}
I20260924 10:12:08.025141 23251 raft_consensus.cc:482] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:12:08.025408 23251 raft_consensus.cc:504] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:12:08.026417 23251 leader_election.cc:288] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735), 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631)
W20260924 10:12:08.027916 22862 leader_election.cc:341] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631): Not found: Tablet not found: 5d916d70fbcd4d32be16b97d08e3bb48
W20260924 10:12:08.028180 22860 leader_election.cc:341] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735): Not found: Tablet not found: 5d916d70fbcd4d32be16b97d08e3bb48
I20260924 10:12:08.028271 22860 leader_election.cc:302] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddbf040cea454ed8bcffb944fcc007f0; no voters: 103da8104d9f496fb4998c589a5db906, 216bcbd68e2c489991d5cc631c7c213c
I20260924 10:12:08.028620 23251 raft_consensus.cc:2726] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:12:08.351866 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:08.351044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:43173)
0924 10:12:08.351621 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:08.351702 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":293,"threads_started":1}
I20260924 10:12:08.793875 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:08.793057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:36399)
0924 10:12:08.793582 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:08.793683 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":286,"threads_started":1}
I20260924 10:12:08.900041 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:08.899284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:51013)
0924 10:12:08.899740 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:08.899864 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260924 10:12:09.353900 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:09.352913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:39237)
0924 10:12:09.353562 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:09.353664 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":328,"threads_started":1}
I20260924 10:12:09.795699 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:09.794844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:34429)
0924 10:12:09.795367 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:09.795457 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":278,"threads_started":1}
I20260924 10:12:09.901882 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:09.901076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:50963)
0924 10:12:09.901570 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:09.901660 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":230,"threads_started":1}
I20260924 10:12:10.355638 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:10.354886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:33631)
0924 10:12:10.355393 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:10.355477 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":291,"threads_started":1}
I20260924 10:12:10.768591 23259 raft_consensus.cc:482] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:12:10.768842 23259 raft_consensus.cc:504] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:12:10.769555 23259 leader_election.cc:288] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735), 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631)
W20260924 10:12:10.770656 22860 leader_election.cc:341] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735): Not found: Tablet not found: 5d916d70fbcd4d32be16b97d08e3bb48
W20260924 10:12:10.770891 22862 leader_election.cc:341] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631): Not found: Tablet not found: 5d916d70fbcd4d32be16b97d08e3bb48
I20260924 10:12:10.771049 22862 leader_election.cc:302] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddbf040cea454ed8bcffb944fcc007f0; no voters: 103da8104d9f496fb4998c589a5db906, 216bcbd68e2c489991d5cc631c7c213c
I20260924 10:12:10.771443 23259 raft_consensus.cc:2726] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:12:10.797387 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:10.796633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:55871)
0924 10:12:10.797124 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:10.797208 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":254,"threads_started":1}
I20260924 10:12:10.903709 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:10.902806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:38913)
0924 10:12:10.903406 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:10.903516 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":314,"threads_started":1}
I20260924 10:12:11.357385 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:11.356625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:35289)
0924 10:12:11.357120 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:11.357203 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
I20260924 10:12:11.799381 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:11.798479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:32953)
0924 10:12:11.799102 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:11.799191 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":414,"threads_started":1}
I20260924 10:12:11.905508 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:11.904670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:52533)
0924 10:12:11.905208 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:11.905300 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":289,"threads_started":1}
I20260924 10:12:12.359688 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:12.358601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:49381)
0924 10:12:12.359422 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:12.359501 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":288,"threads_started":1}
I20260924 10:12:12.801986 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:12.800773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:53135)
0924 10:12:12.801562 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:12.801706 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":381,"threads_started":1}
I20260924 10:12:12.907380 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:12.906562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:45579)
0924 10:12:12.907131 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:12.907216 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":289,"threads_started":1}
I20260924 10:12:13.210439 23268 raft_consensus.cc:482] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:12:13.210654 23268 raft_consensus.cc:504] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:12:13.211483 23268 leader_election.cc:288] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735), 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631)
W20260924 10:12:13.212507 22860 leader_election.cc:341] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735): Not found: Tablet not found: 5d916d70fbcd4d32be16b97d08e3bb48
W20260924 10:12:13.212697 22862 leader_election.cc:341] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631): Not found: Tablet not found: 5d916d70fbcd4d32be16b97d08e3bb48
I20260924 10:12:13.212795 22862 leader_election.cc:302] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddbf040cea454ed8bcffb944fcc007f0; no voters: 103da8104d9f496fb4998c589a5db906, 216bcbd68e2c489991d5cc631c7c213c
I20260924 10:12:13.213063 23268 raft_consensus.cc:2726] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:12:13.361373 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:13.360635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:50695)
0924 10:12:13.361098 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:13.361185 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":246,"threads_started":1}
I20260924 10:12:13.804319 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:13.803377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:46509)
0924 10:12:13.803939 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:13.804052 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":275,"threads_started":1}
I20260924 10:12:13.909081 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:13.908353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:52633)
0924 10:12:13.908783 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:13.908870 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":217,"threads_started":1}
I20260924 10:12:14.363127 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:14.362341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:46293)
0924 10:12:14.362841 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:14.362936 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":247,"threads_started":1}
I20260924 10:12:14.806044 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:14.805297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:58079)
0924 10:12:14.805798 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:14.805890 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":246,"threads_started":1}
I20260924 10:12:14.911244 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:14.910361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:54325)
0924 10:12:14.910862 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:14.910980 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":282,"threads_started":1}
I20260924 10:12:15.364943 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:15.364160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:42745)
0924 10:12:15.364638 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:15.364722 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":256,"threads_started":1}
I20260924 10:12:15.807859 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:15.807116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:56279)
0924 10:12:15.807587 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:15.807670 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":280,"threads_started":1}
I20260924 10:12:15.913196 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:15.912304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:45415)
0924 10:12:15.912876 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:15.913007 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":331,"threads_started":1}
I20260924 10:12:16.367020 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:16.366066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:46701)
0924 10:12:16.366659 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:16.366793 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":335,"threads_started":1}
I20260924 10:12:16.809648 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:16.808853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:33941)
0924 10:12:16.809331 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:16.809419 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":235,"threads_started":1}
I20260924 10:12:16.914985 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:16.914152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:40169)
0924 10:12:16.914701 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:16.914799 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":338,"threads_started":1}
I20260924 10:12:17.368870 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:17.368094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:54745)
0924 10:12:17.368596 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:17.368692 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":255,"threads_started":1}
I20260924 10:12:17.811656 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:17.810800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:38395)
0924 10:12:17.811331 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:17.811410 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":322,"threads_started":1}
I20260924 10:12:17.916910 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:17.916071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:40969)
0924 10:12:17.916622 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:17.916728 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":306,"threads_started":1}
I20260924 10:12:18.370653 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:18.369866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:40725)
0924 10:12:18.370387 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:18.370481 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":265,"threads_started":1}
I20260924 10:12:18.813597 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:18.812743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:41665)
0924 10:12:18.813254 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:18.813359 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":277,"threads_started":1}
I20260924 10:12:18.918732 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:18.917978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:52747)
0924 10:12:18.918468 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:18.918548 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":264,"threads_started":1}
I20260924 10:12:19.372298 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:19.371563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:57493)
0924 10:12:19.372023 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:19.372105 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":229,"threads_started":1}
I20260924 10:12:19.433387 23288 raft_consensus.cc:482] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:12:19.433573 23288 raft_consensus.cc:504] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "103da8104d9f496fb4998c589a5db906" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40735 } } peers { permanent_uuid: "216bcbd68e2c489991d5cc631c7c213c" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 43631 } } peers { permanent_uuid: "ddbf040cea454ed8bcffb944fcc007f0" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42707 } }
I20260924 10:12:19.434183 23288 leader_election.cc:288] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735), 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631)
W20260924 10:12:19.435233 22860 leader_election.cc:341] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 103da8104d9f496fb4998c589a5db906 (127.6.2.194:40735): Not found: Tablet not found: 5d916d70fbcd4d32be16b97d08e3bb48
W20260924 10:12:19.435432 22862 leader_election.cc:341] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 216bcbd68e2c489991d5cc631c7c213c (127.6.2.195:43631): Not found: Tablet not found: 5d916d70fbcd4d32be16b97d08e3bb48
I20260924 10:12:19.435523 22862 leader_election.cc:302] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddbf040cea454ed8bcffb944fcc007f0; no voters: 103da8104d9f496fb4998c589a5db906, 216bcbd68e2c489991d5cc631c7c213c
I20260924 10:12:19.435725 23288 raft_consensus.cc:2726] T 5d916d70fbcd4d32be16b97d08e3bb48 P ddbf040cea454ed8bcffb944fcc007f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:12:19.816071 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:19.814820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:41685)
0924 10:12:19.815643 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:19.815787 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":395,"threads_started":1}
I20260924 10:12:19.921224 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:19.920118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.193:36431)
0924 10:12:19.920787 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:19.920919 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":246,"threads_started":1}
I20260924 10:12:20.374706 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:20.373765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.194:40695)
0924 10:12:20.374386 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:20.374484 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":358,"threads_started":1}
I20260924 10:12:20.790210 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 22802
I20260924 10:12:20.808446 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:20.792062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.6.2.193:42707)
0924 10:12:20.792900 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:20.792911 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:20.792950 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:12:20.793520 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:20.793529 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:20.793906 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:20.794085 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:20.795757 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:20.796748 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:20.798086 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:20.798686 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:20.798793 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:20.801780 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:20.801807 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:20.801829 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:20.801888 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:20.805029 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:20.806119 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:20.806127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:20.806130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:20.806318 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:20.806813 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:20.806817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:20.806819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:20.807284 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:20.807523 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:20.807804 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 10:12:20.808046 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":810,"threads_started":1}
I20260924 10:12:20.809996 22968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:20.818228 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:20.817417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:34197 (local address 127.6.2.195:55203)
0924 10:12:20.817957 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:20.818062 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.254:34197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":290,"threads_started":1}
I20260924 10:12:20.905757 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 22802
I20260924 10:12:20.926611 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 22977
I20260924 10:12:20.943775 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:20.927518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.6.2.194:40735)
0924 10:12:20.928073 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:20.928078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:20.928094 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:12:20.928147 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:20.928151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:20.928209 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:20.928339 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:20.929748 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:20.930671 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:20.932175 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:20.932609 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:20.932747 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:20.936340 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:20.936347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:20.936351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:20.936408 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:20.940112 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:20.941263 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:20.941270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:20.941273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:20.941502 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:20.942016 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:20.942021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:20.942024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:20.942370 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:20.942701 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:20.942975 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 10:12:20.943294 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":299,"threads_started":1}
I20260924 10:12:20.944971 23100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:21.033453 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 22977
I20260924 10:12:21.052745 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 23108
I20260924 10:12:21.067961 23297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:21.053890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57856 (local address 127.6.2.195:43631)
0924 10:12:21.054676 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:21.054681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:21.054699 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:12:21.054777 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:21.054781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:21.054855 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:21.054995 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:21.056376 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:21.057153 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:21.058385 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:21.058816 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:21.058960 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:21.061748 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:21.061753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:21.061756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:21.061792 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:21.065068 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:21.065841 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:21.065855 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:21.065857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:21.066008 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:21.066390 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:21.066396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:21.066399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:21.066668 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:21.066965 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:21.067316 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 10:12:21.067545 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":312,"threads_started":1}
I20260924 10:12:21.068801 23231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:21.156876 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 23108
2026-09-24T10:12:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64893 ms)
[----------] 30 tests from SecurityITest (304071 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:12:21 dist-test-slave-s3n3 krb5kdc[23301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:12:21 dist-test-slave-s3n3 krb5kdc[23301](info): set up 2 sockets
Sep 24 10:12:21 dist-test-slave-s3n3 krb5kdc[23301](info): commencing operation
krb5kdc: starting...
W20260924 10:12:22.291301 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:12:22 dist-test-slave-s3n3 krb5kdc[23301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:12:22.554711 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:46561
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:46561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:22.889811 23314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:22.890116 23314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:22.890204 23314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:22.897635 23314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:22.897778 23314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:22.897833 23314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:22.897927 23314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:22.898001 23314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:22.908352 23314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:46561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:22.909749 23314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:22.911098 23314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:22.923194 23319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:22.923276 23322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:22.923170 23320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:22.924059 23314 server_base.cc:1034] running on GCE node
I20260924 10:12:22.924840 23314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:22.926292 23314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:22.927546 23314 hybrid_clock.cc:648] HybridClock initialized: now 1790244742927497 us; error 59 us; skew 500 ppm
I20260924 10:12:22.927960 23314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:22 dist-test-slave-s3n3 krb5kdc[23301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244742, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:22.931254 23314 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:12:22.932802 23314 webserver.cc:460] Webserver started at http://127.6.2.254:35879/ using document root <none> and password file <none>
I20260924 10:12:22.933517 23314 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:22.933673 23314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:22.933990 23314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:22.937106 23314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "7b384bd73fcc41ca8ed9ade4dfedde92"
format_stamp: "Formatted at 2026-09-24 10:12:22 on dist-test-slave-s3n3"
server_key: "40aa938e04cb58343bb6a09e882fc0e1"
server_key_iv: "e1627dfaa06a338dc4f0032cae54a01c"
server_key_version: "encryptionkey@0"
I20260924 10:12:22.937942 23314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "7b384bd73fcc41ca8ed9ade4dfedde92"
format_stamp: "Formatted at 2026-09-24 10:12:22 on dist-test-slave-s3n3"
server_key: "40aa938e04cb58343bb6a09e882fc0e1"
server_key_iv: "e1627dfaa06a338dc4f0032cae54a01c"
server_key_version: "encryptionkey@0"
I20260924 10:12:22.943605 23314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 10:12:22.947391 23329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:22.948894 23314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:12:22.949117 23314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "7b384bd73fcc41ca8ed9ade4dfedde92"
format_stamp: "Formatted at 2026-09-24 10:12:22 on dist-test-slave-s3n3"
server_key: "40aa938e04cb58343bb6a09e882fc0e1"
server_key_iv: "e1627dfaa06a338dc4f0032cae54a01c"
server_key_version: "encryptionkey@0"
I20260924 10:12:22.949362 23314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:22.971122 23314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:22.975630 23314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:22.975982 23314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:22.995586 23314 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:46561
I20260924 10:12:22.995615 23380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:46561 every 8 connection(s)
I20260924 10:12:22.997064 23314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:23.000383 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 23314
I20260924 10:12:23.000605 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:23.001410 6155 external_mini_cluster.cc:1428] Setting key 6a80b9a42ee1721e119c8ab4a205eacb
I20260924 10:12:23.001693 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:23.017629 23381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92: Bootstrap starting.
Sep 24 10:12:23 dist-test-slave-s3n3 krb5kdc[23301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:12:23.021287 23381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:23.022478 23381 log.cc:824] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:23.025490 23381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92: No bootstrap required, opened a new log
I20260924 10:12:23.027197 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:23.003688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50530 (local address 127.6.2.254:46561)
0924 10:12:23.004670 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:23.004688 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:23.004739 (+ 51us) server_negotiation.cc:407] Connection header received
0924 10:12:23.007339 (+ 2600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:23.007364 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:23.007834 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:23.008208 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:23.010124 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:23.011537 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:23.013442 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:23.014347 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:23.014696 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:23.019755 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:23.019765 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:23.019782 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:23.019834 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:23.023022 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:23.023850 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:23.023861 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:23.023873 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:23.024045 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:23.024434 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:23.024440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:23.024443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:23.024985 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:23.025302 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:23.025729 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 10:12:23.026195 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":446,"threads_started":1}
I20260924 10:12:23.033957 23381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b384bd73fcc41ca8ed9ade4dfedde92" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46561 } }
I20260924 10:12:23.034390 23381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:23.034502 23381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b384bd73fcc41ca8ed9ade4dfedde92, State: Initialized, Role: FOLLOWER
I20260924 10:12:23.035187 23381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b384bd73fcc41ca8ed9ade4dfedde92" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46561 } }
I20260924 10:12:23.035454 23381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:23.035607 23381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:23.035776 23381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:23.038087 23381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b384bd73fcc41ca8ed9ade4dfedde92" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46561 } }
I20260924 10:12:23.038625 23381 leader_election.cc:302] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b384bd73fcc41ca8ed9ade4dfedde92; no voters:
I20260924 10:12:23.039332 23381 leader_election.cc:288] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:23.039507 23386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:23.040297 23386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [term 1 LEADER]: Becoming Leader. State: Replica: 7b384bd73fcc41ca8ed9ade4dfedde92, State: Running, Role: LEADER
I20260924 10:12:23.040890 23386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b384bd73fcc41ca8ed9ade4dfedde92" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46561 } }
I20260924 10:12:23.041625 23381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:23.043732 23387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b384bd73fcc41ca8ed9ade4dfedde92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b384bd73fcc41ca8ed9ade4dfedde92" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46561 } } }
I20260924 10:12:23.044111 23387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:23.049729 23388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b384bd73fcc41ca8ed9ade4dfedde92. Latest consensus state: current_term: 1 leader_uuid: "7b384bd73fcc41ca8ed9ade4dfedde92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b384bd73fcc41ca8ed9ade4dfedde92" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 46561 } } }
I20260924 10:12:23.050123 23388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:23.050856 23391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:23.056239 23391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:23.065742 23391 catalog_manager.cc:1347] Generated new cluster ID: c7c3aff722db4b09821d86f6f1170e3c
I20260924 10:12:23.065898 23391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:23.075793 23391 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:23.077234 23391 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:23.086352 23391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92: Generated new TSK 0
I20260924 10:12:23.087414 23391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:12:23.150889 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46561
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:23.490965 23407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:23.491362 23407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:23.491470 23407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:23.498980 23407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:23.499161 23407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:23.499285 23407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:12:23.509320 23407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:23.510608 23407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:23.511885 23407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:23.524717 23412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:23.525868 23413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:23.526499 23415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:23.527884 23407 server_base.cc:1034] running on GCE node
I20260924 10:12:23.528501 23407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:23.529819 23407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:23.533218 23407 hybrid_clock.cc:648] HybridClock initialized: now 1790244743533161 us; error 56 us; skew 500 ppm
I20260924 10:12:23.533612 23407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:23 dist-test-slave-s3n3 krb5kdc[23301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244743, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:23.537141 23407 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:12:23.538991 23407 webserver.cc:460] Webserver started at http://127.6.2.193:37789/ using document root <none> and password file <none>
I20260924 10:12:23.539808 23407 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:23.539959 23407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:23.540303 23407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:23.543459 23407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "0349e23ae95a4c5e884a59473939c4d3"
format_stamp: "Formatted at 2026-09-24 10:12:23 on dist-test-slave-s3n3"
server_key: "6e79aa5dc2319b9ddea409500b06f628"
server_key_iv: "9e0105ab3a86d37a9f97d3bab76d9889"
server_key_version: "encryptionkey@0"
I20260924 10:12:23.544286 23407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "0349e23ae95a4c5e884a59473939c4d3"
format_stamp: "Formatted at 2026-09-24 10:12:23 on dist-test-slave-s3n3"
server_key: "6e79aa5dc2319b9ddea409500b06f628"
server_key_iv: "9e0105ab3a86d37a9f97d3bab76d9889"
server_key_version: "encryptionkey@0"
I20260924 10:12:23.550637 23407 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 10:12:23.555292 23422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:23.557237 23407 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 10:12:23.557482 23407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "0349e23ae95a4c5e884a59473939c4d3"
format_stamp: "Formatted at 2026-09-24 10:12:23 on dist-test-slave-s3n3"
server_key: "6e79aa5dc2319b9ddea409500b06f628"
server_key_iv: "9e0105ab3a86d37a9f97d3bab76d9889"
server_key_version: "encryptionkey@0"
I20260924 10:12:23.557737 23407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:23.574445 23407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:23.580168 23407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:23.580495 23407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:23.581548 23407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:23.583117 23407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:23.583232 23407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:23.583390 23407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:23.583468 23407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:23.619355 23407 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:33005
I20260924 10:12:23.619407 23534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:33005 every 8 connection(s)
I20260924 10:12:23.620687 23407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:12:23.627763 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 23407
I20260924 10:12:23.628010 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:12:23.628872 6155 external_mini_cluster.cc:1428] Setting key 44538077e81bb1b7f48e237a212cdc02
Sep 24 10:12:23 dist-test-slave-s3n3 krb5kdc[23301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244743, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:12:23.650780 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:23.623302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:47221 (local address 127.6.2.254:46561)
0924 10:12:23.623726 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:23.623731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:23.624726 (+ 995us) server_negotiation.cc:407] Connection header received
0924 10:12:23.626579 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:23.626584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:23.626668 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:23.626769 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:23.628865 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:23.629765 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:23.631833 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:23.632584 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:23.632732 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:23.643985 (+ 11253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:23.643992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:23.644014 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:23.644072 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:23.646964 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:23.648096 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:23.648101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:23.648103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:23.648237 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:23.648764 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:23.648767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:23.648769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:23.649119 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:23.649237 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:23.650294 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0924 10:12:23.650511 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":200,"threads_started":1}
I20260924 10:12:23.652406 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:23.623857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:46561 (local address 127.6.2.193:47221)
0924 10:12:23.624559 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:23.624597 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:23.626274 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:23.626976 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:23.626984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:23.627805 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:23.628654 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:23.628682 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:23.629959 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:23.629965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:23.631642 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:23.631655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:23.632811 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:23.632817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:23.633031 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:23.639975 (+ 6944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:12:23.640024 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:23.643568 (+ 3544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:23.647186 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:23.647201 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:23.647224 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:23.647922 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:23.648367 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:23.648375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:23.648381 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:23.648647 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:23.649263 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:23.649275 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:23.649821 (+ 546us) client_negotiation.cc:769] Sending connection context
0924 10:12:23.650157 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 10:12:23.650778 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":276,"threads_started":1}
I20260924 10:12:23.654049 23535 heartbeater.cc:347] Connected to a master server at 127.6.2.254:46561
I20260924 10:12:23.654410 23535 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:23.655436 23535 heartbeater.cc:511] Master 127.6.2.254:46561 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
I20260924 10:12:23.657963 23346 ts_manager.cc:194] Registered new tserver with Master: 0349e23ae95a4c5e884a59473939c4d3 (127.6.2.193:33005)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
I20260924 10:12:23.659756 23346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:47221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:12:23.693292 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46561
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:24.020015 23541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:24.020293 23541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:24.020442 23541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:24.027869 23541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:24.027978 23541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:24.028160 23541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:12:24.038304 23541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:24.039647 23541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:24.040940 23541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:24.054950 23549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:24.054834 23546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:24.054809 23547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:24.055629 23541 server_base.cc:1034] running on GCE node
I20260924 10:12:24.056249 23541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:24.057247 23541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:24.058501 23541 hybrid_clock.cc:648] HybridClock initialized: now 1790244744058466 us; error 66 us; skew 500 ppm
I20260924 10:12:24.058871 23541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:24 dist-test-slave-s3n3 krb5kdc[23301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244744, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:24.062156 23541 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:12:24.063820 23541 webserver.cc:460] Webserver started at http://127.6.2.194:46843/ using document root <none> and password file <none>
I20260924 10:12:24.064538 23541 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:24.064683 23541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:24.064992 23541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:24.068017 23541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "ebe508e7d7bf4f03986baa10ac517a18"
format_stamp: "Formatted at 2026-09-24 10:12:24 on dist-test-slave-s3n3"
server_key: "3abd76ef6e9eca5512beb7062b56c1b8"
server_key_iv: "aa13041abc3b77651e6f77a977eae3b5"
server_key_version: "encryptionkey@0"
I20260924 10:12:24.068845 23541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "ebe508e7d7bf4f03986baa10ac517a18"
format_stamp: "Formatted at 2026-09-24 10:12:24 on dist-test-slave-s3n3"
server_key: "3abd76ef6e9eca5512beb7062b56c1b8"
server_key_iv: "aa13041abc3b77651e6f77a977eae3b5"
server_key_version: "encryptionkey@0"
I20260924 10:12:24.073989 23541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 10:12:24.077551 23556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:24.078886 23541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:12:24.079137 23541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "ebe508e7d7bf4f03986baa10ac517a18"
format_stamp: "Formatted at 2026-09-24 10:12:24 on dist-test-slave-s3n3"
server_key: "3abd76ef6e9eca5512beb7062b56c1b8"
server_key_iv: "aa13041abc3b77651e6f77a977eae3b5"
server_key_version: "encryptionkey@0"
I20260924 10:12:24.079345 23541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:24.095800 23541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:24.100632 23541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:24.100948 23541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:24.101959 23541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:24.103466 23541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:24.103574 23541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:24.103734 23541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:24.103832 23541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:24.138326 23541 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:38427
I20260924 10:12:24.138360 23668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:38427 every 8 connection(s)
I20260924 10:12:24.139807 23541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:12:24.148695 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 23541
I20260924 10:12:24.148900 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:12:24.149523 6155 external_mini_cluster.cc:1428] Setting key 10975cc544b4e07f38949d2c017ceb92
Sep 24 10:12:24 dist-test-slave-s3n3 krb5kdc[23301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244744, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:12:24.162011 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.142577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:60579 (local address 127.6.2.254:46561)
0924 10:12:24.142768 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:24.142773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:24.143919 (+ 1146us) server_negotiation.cc:407] Connection header received
0924 10:12:24.145735 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:24.145740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:24.145814 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:24.145905 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:24.147444 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.148160 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.150436 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.150940 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.151107 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.155527 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:24.155532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:24.155535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:24.155572 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:24.158513 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:24.159529 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:24.159534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:24.159538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:24.159679 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:24.160131 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:24.160136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:24.160140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:24.160452 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:24.160632 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:24.161370 (+ 738us) server_negotiation.cc:299] Negotiation successful
0924 10:12:24.161632 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:12:24.163053 23671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.143014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:46561 (local address 127.6.2.194:60579)
0924 10:12:24.143758 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:24.143799 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:24.145464 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:24.146133 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:24.146140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:24.146644 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:24.147277 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.147296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.148333 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.148340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.150282 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.150294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.151200 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.151207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.151440 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.152273 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:12:24.152310 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:24.155274 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:24.158697 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:24.158708 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:24.158734 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:24.159390 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:24.159804 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:24.159810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:24.159814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:24.160007 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:24.160596 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:24.160604 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:24.160991 (+ 387us) client_negotiation.cc:769] Sending connection context
0924 10:12:24.161269 (+ 278us) client_negotiation.cc:240] Negotiation successful
0924 10:12:24.161842 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":274,"threads_started":1}
I20260924 10:12:24.164651 23669 heartbeater.cc:347] Connected to a master server at 127.6.2.254:46561
I20260924 10:12:24.164989 23669 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:24.165875 23669 heartbeater.cc:511] Master 127.6.2.254:46561 requested a full tablet report, sending...
I20260924 10:12:24.167665 23346 ts_manager.cc:194] Registered new tserver with Master: ebe508e7d7bf4f03986baa10ac517a18 (127.6.2.194:38427)
I20260924 10:12:24.168951 23346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:60579
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:12:24.211711 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46561
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:24.544304 23674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:24.544607 23674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:24.544763 23674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:24.552179 23674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:24.552302 23674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:24.552513 23674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:12:24.562942 23674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:46561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:24.564301 23674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:24.565667 23674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:24.579398 23680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:24.579342 23682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:24.579432 23679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:24.580255 23674 server_base.cc:1034] running on GCE node
I20260924 10:12:24.580919 23674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:24.581926 23674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:24.583168 23674 hybrid_clock.cc:648] HybridClock initialized: now 1790244744583116 us; error 57 us; skew 500 ppm
I20260924 10:12:24.583518 23674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:24 dist-test-slave-s3n3 krb5kdc[23301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244744, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:24.586947 23674 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:12:24.588675 23674 webserver.cc:460] Webserver started at http://127.6.2.195:46545/ using document root <none> and password file <none>
I20260924 10:12:24.589395 23674 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:24.589540 23674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:24.589856 23674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:24.592849 23674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "48ec9b30440f41cf99cada6db9b25867"
format_stamp: "Formatted at 2026-09-24 10:12:24 on dist-test-slave-s3n3"
server_key: "7343b4a237d5eea6281119207af3cc63"
server_key_iv: "aabe036eeca8fe3de40ed674d79b508e"
server_key_version: "encryptionkey@0"
I20260924 10:12:24.593647 23674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "48ec9b30440f41cf99cada6db9b25867"
format_stamp: "Formatted at 2026-09-24 10:12:24 on dist-test-slave-s3n3"
server_key: "7343b4a237d5eea6281119207af3cc63"
server_key_iv: "aabe036eeca8fe3de40ed674d79b508e"
server_key_version: "encryptionkey@0"
I20260924 10:12:24.598744 23674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260924 10:12:24.602170 23689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:24.603645 23674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:12:24.603816 23674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "48ec9b30440f41cf99cada6db9b25867"
format_stamp: "Formatted at 2026-09-24 10:12:24 on dist-test-slave-s3n3"
server_key: "7343b4a237d5eea6281119207af3cc63"
server_key_iv: "aabe036eeca8fe3de40ed674d79b508e"
server_key_version: "encryptionkey@0"
I20260924 10:12:24.604029 23674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:24.632133 23674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:24.636327 23674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:24.636631 23674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:24.637603 23674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:24.638981 23674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:24.639104 23674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:24.639256 23674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:24.639330 23674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:24.663174 23535 heartbeater.cc:503] Master 127.6.2.254:46561 was elected leader, sending a full tablet report...
I20260924 10:12:24.673609 23674 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:42241
I20260924 10:12:24.673826 23801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:42241 every 8 connection(s)
I20260924 10:12:24.675019 23674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:12:24.676848 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 23674
I20260924 10:12:24.677031 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:12:24.677506 6155 external_mini_cluster.cc:1428] Setting key 59699e881dffc48c023b330a50d9e649
Sep 24 10:12:24 dist-test-slave-s3n3 krb5kdc[23301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244744, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:12:24.696261 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.677766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:60457 (local address 127.6.2.254:46561)
0924 10:12:24.678267 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:24.678271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:24.679270 (+ 999us) server_negotiation.cc:407] Connection header received
0924 10:12:24.681396 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:24.681402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:24.681488 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:24.681618 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:24.683106 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.683895 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.685455 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.685874 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.686008 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.689641 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:24.689646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:24.689650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:24.689685 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:24.693081 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:24.694046 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:24.694050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:24.694053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:24.694188 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:24.694577 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:24.694580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:24.694582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:24.694905 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:24.695032 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:24.695805 (+ 773us) server_negotiation.cc:299] Negotiation successful
0924 10:12:24.695988 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":270,"threads_started":1}
I20260924 10:12:24.697257 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.678341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:46561 (local address 127.6.2.195:60457)
0924 10:12:24.679123 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:24.679175 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:24.681131 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:24.681840 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:24.681849 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:24.682347 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:24.682903 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.682941 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.684049 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.684059 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.685296 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.685308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.686113 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.686121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.686349 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.687009 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:12:24.687034 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:24.689380 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:24.693256 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:24.693266 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:24.693286 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:24.693923 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:24.694287 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:24.694290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:24.694292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:24.694480 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:24.695018 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:24.695024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:24.695436 (+ 412us) client_negotiation.cc:769] Sending connection context
0924 10:12:24.695680 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 10:12:24.696097 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":369,"threads_started":1}
I20260924 10:12:24.698791 23802 heartbeater.cc:347] Connected to a master server at 127.6.2.254:46561
I20260924 10:12:24.699173 23802 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:24.699910 23802 heartbeater.cc:511] Master 127.6.2.254:46561 requested a full tablet report, sending...
I20260924 10:12:24.701498 23346 ts_manager.cc:194] Registered new tserver with Master: 48ec9b30440f41cf99cada6db9b25867 (127.6.2.195:42241)
I20260924 10:12:24.702308 23346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:60457
I20260924 10:12:24.706347 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:12:24.725312 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.711190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.6.2.254:46561)
0924 10:12:24.711436 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:24.711442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:24.711763 (+ 321us) server_negotiation.cc:407] Connection header received
0924 10:12:24.711873 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:24.711877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:24.711944 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:24.712031 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:24.713440 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.714183 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.715613 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.716036 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.716136 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.718620 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:24.718624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:24.718628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:24.718662 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:24.721820 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:24.722688 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:24.722696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:24.722702 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:24.722878 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:24.723407 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:24.723413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:24.723419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:24.723922 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:24.724223 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:24.724625 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 10:12:24.724895 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:12:24.730501 23346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:12:24.737484 23346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:12:24.764570 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.750122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:33005 (local address 127.0.0.1:56378)
0924 10:12:24.753386 (+ 3264us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:24.753441 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:24.753704 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:24.754593 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:24.754600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:24.754631 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:24.754997 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.755010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.757365 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.757374 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.760603 (+ 3229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.760625 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.763472 (+ 2847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.763480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.763708 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.763768 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 10:12:24.763901 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 10:12:24.764146 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2985,"thread_start_us":3821,"threads_started":1}
I20260924 10:12:24.765394 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.750212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.6.2.193:33005)
0924 10:12:24.753137 (+ 2925us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:24.753146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:24.753551 (+ 405us) server_negotiation.cc:407] Connection header received
0924 10:12:24.753960 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:24.753969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:24.754294 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:24.754552 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:24.755256 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.757147 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.760888 (+ 3741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.763203 (+ 2315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.764744 (+ 1541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.764975 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:24.765047 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 10:12:24.765172 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2749,"thread_start_us":2588,"threads_started":1}
I20260924 10:12:24.766646 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.750527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:42241 (local address 127.0.0.1:57160)
0924 10:12:24.754145 (+ 3618us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:24.754162 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:24.754302 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:24.755596 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:24.755600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:24.755624 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:24.755996 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.756008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.759880 (+ 3872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.759888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.763894 (+ 4006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.763908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.765820 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.765829 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.766037 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.766071 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:12:24.766160 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 10:12:24.766291 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3516,"thread_start_us":441,"threads_started":1}
I20260924 10:12:24.767400 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.751211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57160 (local address 127.6.2.195:42241)
0924 10:12:24.753671 (+ 2460us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:24.753678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:24.754261 (+ 583us) server_negotiation.cc:407] Connection header received
0924 10:12:24.754461 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:24.754467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:24.755142 (+ 675us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:24.755317 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:24.756195 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.758304 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.764097 (+ 5793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.765476 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.765598 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.765829 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:24.766941 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0924 10:12:24.767161 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2288,"thread_start_us":2084,"threads_started":1}
I20260924 10:12:24.769738 23470 tablet_service.cc:1467] Processing CreateTablet for tablet a70b9e0048514c90b83280c68779cf15 (DEFAULT_TABLE table=test-table [id=6671bacad9cd40579208ce76801f1203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:24.771339 23470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70b9e0048514c90b83280c68779cf15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:24.770967 23737 tablet_service.cc:1467] Processing CreateTablet for tablet a70b9e0048514c90b83280c68779cf15 (DEFAULT_TABLE table=test-table [id=6671bacad9cd40579208ce76801f1203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:24.771962 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.756256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:38427 (local address 127.0.0.1:53428)
0924 10:12:24.764250 (+ 7994us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:24.764284 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:24.764411 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:24.765568 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:24.765572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:24.765598 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:24.765993 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.766002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.767818 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.767823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.769722 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.769734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.771159 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.771166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.771336 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.771378 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 10:12:24.771492 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 10:12:24.771636 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7881,"thread_start_us":8244,"threads_started":1}
I20260924 10:12:24.772289 23737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70b9e0048514c90b83280c68779cf15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:24.772382 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.754576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.6.2.194:38427)
0924 10:12:24.763816 (+ 9240us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:24.763824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:24.764408 (+ 584us) server_negotiation.cc:407] Connection header received
0924 10:12:24.764561 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:24.764569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:24.765103 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:24.765310 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:24.766170 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.767656 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.769900 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.770926 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.771100 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.771298 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:24.771668 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 10:12:24.771840 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9079,"thread_start_us":8939,"threads_started":1}
I20260924 10:12:24.775529 23604 tablet_service.cc:1467] Processing CreateTablet for tablet a70b9e0048514c90b83280c68779cf15 (DEFAULT_TABLE table=test-table [id=6671bacad9cd40579208ce76801f1203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:24.776616 23604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70b9e0048514c90b83280c68779cf15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:24.786382 23821 tablet_bootstrap.cc:492] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3: Bootstrap starting.
I20260924 10:12:24.788745 23822 tablet_bootstrap.cc:492] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867: Bootstrap starting.
I20260924 10:12:24.791443 23821 tablet_bootstrap.cc:654] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:24.791747 23823 tablet_bootstrap.cc:492] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18: Bootstrap starting.
I20260924 10:12:24.792843 23821 log.cc:824] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:24.794214 23822 tablet_bootstrap.cc:654] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:24.794735 23823 tablet_bootstrap.cc:654] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:24.795473 23822 log.cc:824] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:24.795750 23823 log.cc:824] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:24.796353 23821 tablet_bootstrap.cc:492] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3: No bootstrap required, opened a new log
I20260924 10:12:24.796895 23821 ts_tablet_manager.cc:1397] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 10:12:24.798739 23822 tablet_bootstrap.cc:492] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867: No bootstrap required, opened a new log
I20260924 10:12:24.799026 23823 tablet_bootstrap.cc:492] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18: No bootstrap required, opened a new log
I20260924 10:12:24.799012 23822 ts_tablet_manager.cc:1397] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 10:12:24.799289 23823 ts_tablet_manager.cc:1397] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260924 10:12:24.805080 23821 raft_consensus.cc:348] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349e23ae95a4c5e884a59473939c4d3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 33005 } } peers { permanent_uuid: "48ec9b30440f41cf99cada6db9b25867" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42241 } } peers { permanent_uuid: "ebe508e7d7bf4f03986baa10ac517a18" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38427 } }
I20260924 10:12:24.805486 23821 raft_consensus.cc:374] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:24.805580 23821 raft_consensus.cc:729] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0349e23ae95a4c5e884a59473939c4d3, State: Initialized, Role: FOLLOWER
I20260924 10:12:24.806197 23821 consensus_queue.cc:260] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349e23ae95a4c5e884a59473939c4d3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 33005 } } peers { permanent_uuid: "48ec9b30440f41cf99cada6db9b25867" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42241 } } peers { permanent_uuid: "ebe508e7d7bf4f03986baa10ac517a18" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38427 } }
I20260924 10:12:24.808012 23821 ts_tablet_manager.cc:1428] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 10:12:24.809953 23823 raft_consensus.cc:348] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349e23ae95a4c5e884a59473939c4d3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 33005 } } peers { permanent_uuid: "48ec9b30440f41cf99cada6db9b25867" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42241 } } peers { permanent_uuid: "ebe508e7d7bf4f03986baa10ac517a18" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38427 } }
I20260924 10:12:24.809952 23822 raft_consensus.cc:348] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349e23ae95a4c5e884a59473939c4d3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 33005 } } peers { permanent_uuid: "48ec9b30440f41cf99cada6db9b25867" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42241 } } peers { permanent_uuid: "ebe508e7d7bf4f03986baa10ac517a18" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38427 } }
I20260924 10:12:24.810392 23822 raft_consensus.cc:374] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:24.810382 23823 raft_consensus.cc:374] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:24.810491 23823 raft_consensus.cc:729] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe508e7d7bf4f03986baa10ac517a18, State: Initialized, Role: FOLLOWER
I20260924 10:12:24.810491 23822 raft_consensus.cc:729] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ec9b30440f41cf99cada6db9b25867, State: Initialized, Role: FOLLOWER
I20260924 10:12:24.811123 23822 consensus_queue.cc:260] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349e23ae95a4c5e884a59473939c4d3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 33005 } } peers { permanent_uuid: "48ec9b30440f41cf99cada6db9b25867" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42241 } } peers { permanent_uuid: "ebe508e7d7bf4f03986baa10ac517a18" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38427 } }
I20260924 10:12:24.811158 23823 consensus_queue.cc:260] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349e23ae95a4c5e884a59473939c4d3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 33005 } } peers { permanent_uuid: "48ec9b30440f41cf99cada6db9b25867" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42241 } } peers { permanent_uuid: "ebe508e7d7bf4f03986baa10ac517a18" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38427 } }
I20260924 10:12:24.812286 23802 heartbeater.cc:503] Master 127.6.2.254:46561 was elected leader, sending a full tablet report...
I20260924 10:12:24.812762 23669 heartbeater.cc:503] Master 127.6.2.254:46561 was elected leader, sending a full tablet report...
I20260924 10:12:24.813264 23823 ts_tablet_manager.cc:1428] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260924 10:12:24.814651 23822 ts_tablet_manager.cc:1428] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
W20260924 10:12:24.873183 23536 tablet.cc:2369] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:12:24.891949 23670 tablet.cc:2369] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:12:24.926704 23803 tablet.cc:2369] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:12:24.976635 23829 raft_consensus.cc:482] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:12:24.976995 23829 raft_consensus.cc:504] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349e23ae95a4c5e884a59473939c4d3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 33005 } } peers { permanent_uuid: "48ec9b30440f41cf99cada6db9b25867" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42241 } } peers { permanent_uuid: "ebe508e7d7bf4f03986baa10ac517a18" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38427 } }
I20260924 10:12:24.978799 23829 leader_election.cc:288] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0349e23ae95a4c5e884a59473939c4d3 (127.6.2.193:33005), 48ec9b30440f41cf99cada6db9b25867 (127.6.2.195:42241)
I20260924 10:12:24.986063 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.979555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:36107 (local address 127.6.2.195:42241)
0924 10:12:24.979799 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:24.979808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:24.980444 (+ 636us) server_negotiation.cc:407] Connection header received
0924 10:12:24.980653 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:24.980657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:24.980716 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:24.980821 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:24.981440 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.982282 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.984240 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.985016 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.985190 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.985303 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:24.985667 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 10:12:24.985796 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 10:12:24.986086 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.979394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:55535 (local address 127.6.2.193:33005)
0924 10:12:24.979610 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:24.979619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:24.980341 (+ 722us) server_negotiation.cc:407] Connection header received
0924 10:12:24.980551 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:24.980555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:24.980616 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:24.980712 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:24.981280 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.982011 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.984224 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.985002 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.985167 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.985286 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:24.985679 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 10:12:24.985812 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:12:24.986259 23830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.979333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:33005 (local address 127.6.2.194:55535)
0924 10:12:24.980205 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:24.980229 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:24.980386 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:24.980739 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:24.980744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:24.980768 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:24.981132 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.981141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.982138 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.982146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.984071 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.984082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.985253 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.985260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.985517 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.985540 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:12:24.985651 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 10:12:24.985774 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":62,"thread_start_us":458,"threads_started":1}
I20260924 10:12:24.986650 23831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:24.979454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:42241 (local address 127.6.2.194:36107)
0924 10:12:24.980328 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:24.980347 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:24.980512 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:24.980838 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:24.980842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:24.980880 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:24.981283 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.981292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.982416 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.982423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.984070 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:24.984082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:24.985292 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:24.985298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:24.985502 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:24.985526 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:12:24.985638 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 10:12:24.985771 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":97,"thread_start_us":533,"threads_started":1}
I20260924 10:12:24.987484 23490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70b9e0048514c90b83280c68779cf15" candidate_uuid: "ebe508e7d7bf4f03986baa10ac517a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0349e23ae95a4c5e884a59473939c4d3" is_pre_election: true
I20260924 10:12:24.987730 23757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70b9e0048514c90b83280c68779cf15" candidate_uuid: "ebe508e7d7bf4f03986baa10ac517a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ec9b30440f41cf99cada6db9b25867" is_pre_election: true
I20260924 10:12:24.987994 23490 raft_consensus.cc:2454] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebe508e7d7bf4f03986baa10ac517a18 in term 0.
I20260924 10:12:24.988276 23757 raft_consensus.cc:2454] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebe508e7d7bf4f03986baa10ac517a18 in term 0.
I20260924 10:12:24.988792 23559 leader_election.cc:302] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0349e23ae95a4c5e884a59473939c4d3, ebe508e7d7bf4f03986baa10ac517a18; no voters:
I20260924 10:12:24.989284 23829 raft_consensus.cc:2781] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:12:24.989446 23829 raft_consensus.cc:482] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:12:24.989552 23829 raft_consensus.cc:3037] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:24.991993 23829 raft_consensus.cc:504] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349e23ae95a4c5e884a59473939c4d3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 33005 } } peers { permanent_uuid: "48ec9b30440f41cf99cada6db9b25867" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42241 } } peers { permanent_uuid: "ebe508e7d7bf4f03986baa10ac517a18" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38427 } }
I20260924 10:12:24.992560 23829 leader_election.cc:288] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [CANDIDATE]: Term 1 election: Requested vote from peers 0349e23ae95a4c5e884a59473939c4d3 (127.6.2.193:33005), 48ec9b30440f41cf99cada6db9b25867 (127.6.2.195:42241)
I20260924 10:12:24.993044 23490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70b9e0048514c90b83280c68779cf15" candidate_uuid: "ebe508e7d7bf4f03986baa10ac517a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0349e23ae95a4c5e884a59473939c4d3"
I20260924 10:12:24.993067 23757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70b9e0048514c90b83280c68779cf15" candidate_uuid: "ebe508e7d7bf4f03986baa10ac517a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ec9b30440f41cf99cada6db9b25867"
I20260924 10:12:24.993260 23757 raft_consensus.cc:3037] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:24.993259 23490 raft_consensus.cc:3037] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:24.995648 23490 raft_consensus.cc:2454] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebe508e7d7bf4f03986baa10ac517a18 in term 1.
I20260924 10:12:24.995685 23757 raft_consensus.cc:2454] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebe508e7d7bf4f03986baa10ac517a18 in term 1.
I20260924 10:12:24.996093 23559 leader_election.cc:302] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0349e23ae95a4c5e884a59473939c4d3, ebe508e7d7bf4f03986baa10ac517a18; no voters:
I20260924 10:12:24.996382 23829 raft_consensus.cc:2781] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:24.996861 23829 raft_consensus.cc:686] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [term 1 LEADER]: Becoming Leader. State: Replica: ebe508e7d7bf4f03986baa10ac517a18, State: Running, Role: LEADER
I20260924 10:12:24.997386 23829 consensus_queue.cc:237] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349e23ae95a4c5e884a59473939c4d3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 33005 } } peers { permanent_uuid: "48ec9b30440f41cf99cada6db9b25867" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42241 } } peers { permanent_uuid: "ebe508e7d7bf4f03986baa10ac517a18" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38427 } }
I20260924 10:12:25.002681 23346 catalog_manager.cc:5495] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebe508e7d7bf4f03986baa10ac517a18 (127.6.2.194). New cstate: current_term: 1 leader_uuid: "ebe508e7d7bf4f03986baa10ac517a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349e23ae95a4c5e884a59473939c4d3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 33005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48ec9b30440f41cf99cada6db9b25867" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebe508e7d7bf4f03986baa10ac517a18" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 38427 } health_report { overall_health: HEALTHY } } }
I20260924 10:12:25.060283 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:25.054652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.6.2.194:38427)
0924 10:12:25.054886 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:25.054892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:25.055016 (+ 124us) server_negotiation.cc:407] Connection header received
0924 10:12:25.055312 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:25.055316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:25.055412 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:25.055572 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:12:25.055951 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:25.056686 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:25.058028 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:25.058452 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:25.058594 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:25.059138 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:12:25.059675 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:12:25.059851 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:25.059903 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 10:12:25.059997 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 10:12:25.068833 23757 raft_consensus.cc:1261] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867 [term 1 FOLLOWER]: Refusing update from remote peer ebe508e7d7bf4f03986baa10ac517a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:12:25.068833 23490 raft_consensus.cc:1261] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3 [term 1 FOLLOWER]: Refusing update from remote peer ebe508e7d7bf4f03986baa10ac517a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:12:25.069911 23832 consensus_queue.cc:1035] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ec9b30440f41cf99cada6db9b25867" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:12:25.070309 23829 consensus_queue.cc:1035] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0349e23ae95a4c5e884a59473939c4d3" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 33005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:12:25.081833 23835 mvcc.cc:205] Tried to move back new op lower bound from 7332842475790913536 to 7332842475513442304. Current Snapshot: MvccSnapshot[applied={T|T < 7332842475790913536}]
I20260924 10:12:25.096530 23344 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:12:25.096817 23344 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:12:25.100524 23344 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b384bd73fcc41ca8ed9ade4dfedde92: Sending DeleteTablet for 3 replicas of tablet a70b9e0048514c90b83280c68779cf15
I20260924 10:12:25.101824 23604 tablet_service.cc:1514] Processing DeleteTablet for tablet a70b9e0048514c90b83280c68779cf15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:25 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:53428
I20260924 10:12:25.101835 23470 tablet_service.cc:1514] Processing DeleteTablet for tablet a70b9e0048514c90b83280c68779cf15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:25 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:56378
I20260924 10:12:25.102558 23737 tablet_service.cc:1514] Processing DeleteTablet for tablet a70b9e0048514c90b83280c68779cf15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:25 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:57160
I20260924 10:12:25.103955 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 23407
I20260924 10:12:25.113372 23847 tablet_replica.cc:330] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867: stopping tablet replica
I20260924 10:12:25.113996 23847 raft_consensus.cc:2229] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:12:25.114676 23847 raft_consensus.cc:2258] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:25.115634 23846 tablet_replica.cc:330] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3: stopping tablet replica
I20260924 10:12:25.115890 23845 tablet_replica.cc:330] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18: stopping tablet replica
I20260924 10:12:25.116060 23846 raft_consensus.cc:2229] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:12:25.116297 23845 raft_consensus.cc:2229] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:12:25.116559 23846 raft_consensus.cc:2258] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:25.117026 23845 raft_consensus.cc:2258] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:25.119019 23846 ts_tablet_manager.cc:1905] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:25.119782 23845 ts_tablet_manager.cc:1905] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:25.124332 23847 ts_tablet_manager.cc:1905] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 10:12:25 dist-test-slave-s3n3 krb5kdc[23301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:12:25.127097 23845 ts_tablet_manager.cc:1918] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:25.127259 23845 log.cc:1196] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/a70b9e0048514c90b83280c68779cf15
I20260924 10:12:25.127799 23845 ts_tablet_manager.cc:1939] T a70b9e0048514c90b83280c68779cf15 P ebe508e7d7bf4f03986baa10ac517a18: Deleting consensus metadata
I20260924 10:12:25.130535 23332 catalog_manager.cc:4841] TS ebe508e7d7bf4f03986baa10ac517a18 (127.6.2.194:38427): tablet a70b9e0048514c90b83280c68779cf15 (table test-table [id=6671bacad9cd40579208ce76801f1203]) successfully deleted
I20260924 10:12:25.131237 23847 ts_tablet_manager.cc:1918] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:25.131412 23847 log.cc:1196] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/a70b9e0048514c90b83280c68779cf15
I20260924 10:12:25.131882 23847 ts_tablet_manager.cc:1939] T a70b9e0048514c90b83280c68779cf15 P 48ec9b30440f41cf99cada6db9b25867: Deleting consensus metadata
I20260924 10:12:25.133649 23846 ts_tablet_manager.cc:1918] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:25.133675 23331 catalog_manager.cc:4841] TS 48ec9b30440f41cf99cada6db9b25867 (127.6.2.195:42241): tablet a70b9e0048514c90b83280c68779cf15 (table test-table [id=6671bacad9cd40579208ce76801f1203]) successfully deleted
I20260924 10:12:25.133817 23846 log.cc:1196] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/a70b9e0048514c90b83280c68779cf15
I20260924 10:12:25.134343 23846 ts_tablet_manager.cc:1939] T a70b9e0048514c90b83280c68779cf15 P 0349e23ae95a4c5e884a59473939c4d3: Deleting consensus metadata
I20260924 10:12:25.136070 23332 catalog_manager.cc:4841] TS 0349e23ae95a4c5e884a59473939c4d3 (127.6.2.193:33005): tablet a70b9e0048514c90b83280c68779cf15 (table test-table [id=6671bacad9cd40579208ce76801f1203]) successfully deleted
I20260924 10:12:25.137084 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:25.107293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.6.2.193:33005)
0924 10:12:25.107535 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:25.107544 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:25.112997 (+ 5453us) server_negotiation.cc:407] Connection header received
0924 10:12:25.113965 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:25.113972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:25.114062 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:25.114138 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:25.116961 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:25.118193 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:25.121114 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:25.121783 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:25.121906 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:25.128078 (+ 6172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:25.128097 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:25.128117 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:25.128171 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:25.133077 (+ 4906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:25.134372 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:25.134382 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:25.134387 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:25.134566 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:25.135054 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:25.135086 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:25.135093 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:25.135600 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:25.136037 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:25.136598 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 10:12:25.136764 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 10:12:25.137780 23530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:25.232914 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 23407
I20260924 10:12:25.252451 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 23541
Sep 24 10:12:25 dist-test-slave-s3n3 krb5kdc[23301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:12:25.269076 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:25.253255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.6.2.194:38427)
0924 10:12:25.253461 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:25.253467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:25.253583 (+ 116us) server_negotiation.cc:407] Connection header received
0924 10:12:25.253795 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:25.253823 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:25.253918 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:25.254032 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:25.255459 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:25.256292 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:25.257632 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:25.258087 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:25.258229 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:25.262627 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:25.262642 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:25.262658 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:25.262693 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:25.265903 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:25.266628 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:25.266636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:25.266638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:25.266774 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:25.267421 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:25.267428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:25.267431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:25.267813 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:25.268179 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:25.268477 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 10:12:25.268705 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:12:25.269773 23664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:25.385703 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 23541
I20260924 10:12:25.404383 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 23674
Sep 24 10:12:25 dist-test-slave-s3n3 krb5kdc[23301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:12:25.419817 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:25.405274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.6.2.195:42241)
0924 10:12:25.405466 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:25.405472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:25.405501 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:12:25.405677 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:25.405681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:25.405746 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:25.405833 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:25.407156 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:25.407961 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:25.409250 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:25.409633 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:25.409736 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:25.414166 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:25.414180 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:25.414196 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:25.414238 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:25.416879 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:25.417583 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:25.417596 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:25.417598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:25.417750 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:25.418135 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:25.418141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:25.418144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:25.418498 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:25.418804 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:25.419317 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 10:12:25.419532 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:12:25.420445 23797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:25.511894 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 23674
I20260924 10:12:25.528667 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 23314
I20260924 10:12:25.529793 23376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:25.641264 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 23314
2026-09-24T10:12:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4488 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T10:12:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:25Z Disabled control of system clock
I20260924 10:12:25.722328 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36177
--webserver_interface=127.6.2.254
--webserver_port=0
--builtin_ntp_servers=127.6.2.212:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:36177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:12:26.055109 23856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:26.055601 23856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:26.055764 23856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:26.063556 23856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:26.063678 23856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:26.063766 23856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:26.063843 23856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:26.063928 23856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:26.073377 23856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:36177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:26.074725 23856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:26.076566 23856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:26.088617 23864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:26.088749 23861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:26.088661 23862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:26.089711 23856 server_base.cc:1034] running on GCE node
I20260924 10:12:26.090569 23856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:26.092083 23856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:26.093313 23856 hybrid_clock.cc:648] HybridClock initialized: now 1790244746093269 us; error 40 us; skew 500 ppm
I20260924 10:12:26.093678 23856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:12:26.096113 23856 webserver.cc:460] Webserver started at http://127.6.2.254:36181/ using document root <none> and password file <none>
I20260924 10:12:26.096800 23856 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:26.096938 23856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:26.097261 23856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:26.100241 23856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "b2fdccfd15f94fa88b196ed4edb1db21"
format_stamp: "Formatted at 2026-09-24 10:12:26 on dist-test-slave-s3n3"
server_key: "630bd030bb47200278eda755ff6e4894"
server_key_iv: "8e44f611fa5e08b28002b532693b93bd"
server_key_version: "encryptionkey@0"
I20260924 10:12:26.101087 23856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "b2fdccfd15f94fa88b196ed4edb1db21"
format_stamp: "Formatted at 2026-09-24 10:12:26 on dist-test-slave-s3n3"
server_key: "630bd030bb47200278eda755ff6e4894"
server_key_iv: "8e44f611fa5e08b28002b532693b93bd"
server_key_version: "encryptionkey@0"
I20260924 10:12:26.106235 23856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:12:26.109690 23870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:26.110992 23856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:12:26.111202 23856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "b2fdccfd15f94fa88b196ed4edb1db21"
format_stamp: "Formatted at 2026-09-24 10:12:26 on dist-test-slave-s3n3"
server_key: "630bd030bb47200278eda755ff6e4894"
server_key_iv: "8e44f611fa5e08b28002b532693b93bd"
server_key_version: "encryptionkey@0"
I20260924 10:12:26.111428 23856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:26.141721 23856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:26.142627 23856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:26.142946 23856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:26.162072 23856 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:36177
I20260924 10:12:26.162091 23921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:36177 every 8 connection(s)
I20260924 10:12:26.163518 23856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:26.166493 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 23856
I20260924 10:12:26.166700 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:26.167207 6155 external_mini_cluster.cc:1428] Setting key 4921fa1a916d0a2852c78d7fd54462be
I20260924 10:12:26.167907 23922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:26.182858 23925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:26.169392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.6.2.254:36177)
0924 10:12:26.170752 (+ 1360us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:26.170768 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:26.170830 (+ 62us) server_negotiation.cc:407] Connection header received
0924 10:12:26.173382 (+ 2552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:26.173411 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:26.173982 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:26.174470 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:26.175667 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:26.177006 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:26.178427 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:26.179170 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:26.179380 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:26.180614 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:26.180621 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:26.180635 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:26.181186 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:26.181265 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:26.181289 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:26.181437 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:26.181619 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 10:12:26.181936 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":470,"threads_started":1}
I20260924 10:12:26.184235 23922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21: Bootstrap starting.
I20260924 10:12:26.187752 23922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:26.188836 23922 log.cc:824] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:26.192373 23922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21: No bootstrap required, opened a new log
I20260924 10:12:26.200316 23922 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fdccfd15f94fa88b196ed4edb1db21" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36177 } }
I20260924 10:12:26.200712 23922 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:26.200814 23922 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2fdccfd15f94fa88b196ed4edb1db21, State: Initialized, Role: FOLLOWER
I20260924 10:12:26.201500 23922 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fdccfd15f94fa88b196ed4edb1db21" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36177 } }
I20260924 10:12:26.201787 23922 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:26.201942 23922 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:26.202140 23922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:26.204391 23922 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fdccfd15f94fa88b196ed4edb1db21" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36177 } }
I20260924 10:12:26.204917 23922 leader_election.cc:302] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2fdccfd15f94fa88b196ed4edb1db21; no voters:
I20260924 10:12:26.205475 23922 leader_election.cc:288] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:26.205636 23927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:26.206732 23927 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [term 1 LEADER]: Becoming Leader. State: Replica: b2fdccfd15f94fa88b196ed4edb1db21, State: Running, Role: LEADER
I20260924 10:12:26.207367 23927 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fdccfd15f94fa88b196ed4edb1db21" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36177 } }
I20260924 10:12:26.207794 23922 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:26.210677 23929 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2fdccfd15f94fa88b196ed4edb1db21. Latest consensus state: current_term: 1 leader_uuid: "b2fdccfd15f94fa88b196ed4edb1db21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fdccfd15f94fa88b196ed4edb1db21" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36177 } } }
I20260924 10:12:26.210647 23928 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2fdccfd15f94fa88b196ed4edb1db21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2fdccfd15f94fa88b196ed4edb1db21" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36177 } } }
I20260924 10:12:26.211148 23929 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:26.211186 23928 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:26.213400 23936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:26.218289 23936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:26.226712 23936 catalog_manager.cc:1347] Generated new cluster ID: 5db8315595234ae89b2f9d0e283b794c
I20260924 10:12:26.226836 23936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:26.245054 23936 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:26.246593 23936 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:26.255493 23936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21: Generated new TSK 0
I20260924 10:12:26.256557 23936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:12:26.272071 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36177
--builtin_ntp_servers=127.6.2.212:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:12:26.604236 23946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:26.604903 23946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:26.605039 23946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:26.614212 23946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:26.614346 23946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:26.614473 23946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:12:26.624354 23946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:26.625686 23946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:26.627538 23946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:26.640798 23954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:26.640799 23952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:26.641106 23951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:26.641690 23946 server_base.cc:1034] running on GCE node
I20260924 10:12:26.642307 23946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:26.643338 23946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:26.644579 23946 hybrid_clock.cc:648] HybridClock initialized: now 1790244746644529 us; error 44 us; skew 500 ppm
I20260924 10:12:26.644902 23946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:12:26.647465 23946 webserver.cc:460] Webserver started at http://127.6.2.193:37437/ using document root <none> and password file <none>
I20260924 10:12:26.648164 23946 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:26.648298 23946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:26.648631 23946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:26.651549 23946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "b9dd373e9f63484e8001c97e412b427c"
format_stamp: "Formatted at 2026-09-24 10:12:26 on dist-test-slave-s3n3"
server_key: "52698ead55b56ffbfc8addad6b314d53"
server_key_iv: "58686731b48dc08857842f11b50a4e81"
server_key_version: "encryptionkey@0"
I20260924 10:12:26.652340 23946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "b9dd373e9f63484e8001c97e412b427c"
format_stamp: "Formatted at 2026-09-24 10:12:26 on dist-test-slave-s3n3"
server_key: "52698ead55b56ffbfc8addad6b314d53"
server_key_iv: "58686731b48dc08857842f11b50a4e81"
server_key_version: "encryptionkey@0"
I20260924 10:12:26.657469 23946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:12:26.661019 23960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:26.662362 23946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:12:26.662564 23946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "b9dd373e9f63484e8001c97e412b427c"
format_stamp: "Formatted at 2026-09-24 10:12:26 on dist-test-slave-s3n3"
server_key: "52698ead55b56ffbfc8addad6b314d53"
server_key_iv: "58686731b48dc08857842f11b50a4e81"
server_key_version: "encryptionkey@0"
I20260924 10:12:26.662796 23946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:26.692929 23946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:26.693835 23946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:26.694114 23946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:26.695194 23946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:26.696779 23946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:26.696900 23946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:26.697039 23946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:26.697124 23946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:26.735191 23946 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:43405
I20260924 10:12:26.735282 24072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:43405 every 8 connection(s)
I20260924 10:12:26.737121 23946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:12:26.747548 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 23946
I20260924 10:12:26.747746 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:12:26.748270 6155 external_mini_cluster.cc:1428] Setting key 7843a4877f9f45d1d6a0f787411b6779
I20260924 10:12:26.750823 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:26.740041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:60275 (local address 127.6.2.254:36177)
0924 10:12:26.740522 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:26.740526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:26.741679 (+ 1153us) server_negotiation.cc:407] Connection header received
0924 10:12:26.743635 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:26.743641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:26.743740 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:26.743883 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:26.744944 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:26.745694 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:26.747405 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:26.747963 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:26.748087 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:26.749951 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:26.749957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:26.749960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:26.749977 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:26.750019 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:26.750032 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:26.750135 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:26.750376 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 10:12:26.750493 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":240,"threads_started":1}
I20260924 10:12:26.751523 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36177
--builtin_ntp_servers=127.6.2.212:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:12:26.752038 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:26.740636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36177 (local address 127.6.2.193:60275)
0924 10:12:26.741515 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:26.741551 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:26.743357 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:26.744122 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:26.744131 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:26.744216 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:26.744764 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:26.744780 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:26.745860 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:26.745865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:26.747226 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:26.747244 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:26.748203 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:26.748210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:26.748524 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:26.749529 (+ 1005us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:12:26.749532 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:26.749595 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:12:26.749706 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:26.750133 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:26.750146 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:26.750156 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:12:26.750374 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 10:12:26.750828 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":277,"threads_started":1}
I20260924 10:12:26.754177 24073 heartbeater.cc:347] Connected to a master server at 127.6.2.254:36177
I20260924 10:12:26.754787 24073 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:26.755728 24073 heartbeater.cc:511] Master 127.6.2.254:36177 requested a full tablet report, sending...
I20260924 10:12:26.757918 23887 ts_manager.cc:194] Registered new tserver with Master: b9dd373e9f63484e8001c97e412b427c (127.6.2.193:43405)
I20260924 10:12:26.759498 23887 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.2.193:60275
W20260924 10:12:27.079995 24077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:27.080341 24077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:27.080497 24077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:27.087834 24077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:27.087978 24077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:27.088169 24077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:12:27.097497 24077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:27.098843 24077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:27.100948 24077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:27.114472 24082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:27.114403 24083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:27.114427 24085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:27.115343 24077 server_base.cc:1034] running on GCE node
I20260924 10:12:27.115953 24077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:27.116923 24077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:27.118135 24077 hybrid_clock.cc:648] HybridClock initialized: now 1790244747118075 us; error 56 us; skew 500 ppm
I20260924 10:12:27.118474 24077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:12:27.120841 24077 webserver.cc:460] Webserver started at http://127.6.2.194:36335/ using document root <none> and password file <none>
I20260924 10:12:27.121528 24077 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:27.121639 24077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:27.121978 24077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:27.124871 24077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "1fcf43e0c68b4067bda473f44be13699"
format_stamp: "Formatted at 2026-09-24 10:12:27 on dist-test-slave-s3n3"
server_key: "b250b30b0834238fe29e9fc229cbbc0d"
server_key_iv: "01dcd4e6dcd1c6d82debefcc1dc10671"
server_key_version: "encryptionkey@0"
I20260924 10:12:27.125725 24077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "1fcf43e0c68b4067bda473f44be13699"
format_stamp: "Formatted at 2026-09-24 10:12:27 on dist-test-slave-s3n3"
server_key: "b250b30b0834238fe29e9fc229cbbc0d"
server_key_iv: "01dcd4e6dcd1c6d82debefcc1dc10671"
server_key_version: "encryptionkey@0"
I20260924 10:12:27.130718 24077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:12:27.134161 24091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:27.135691 24077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:12:27.135895 24077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "1fcf43e0c68b4067bda473f44be13699"
format_stamp: "Formatted at 2026-09-24 10:12:27 on dist-test-slave-s3n3"
server_key: "b250b30b0834238fe29e9fc229cbbc0d"
server_key_iv: "01dcd4e6dcd1c6d82debefcc1dc10671"
server_key_version: "encryptionkey@0"
I20260924 10:12:27.136117 24077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:27.156877 24077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:27.157804 24077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:27.158110 24077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:27.159196 24077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:27.160832 24077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:27.160923 24077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:27.161048 24077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:27.161141 24077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:27.212297 24077 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:41567
I20260924 10:12:27.212412 24203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:41567 every 8 connection(s)
I20260924 10:12:27.213569 24077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:12:27.217414 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 24077
I20260924 10:12:27.217588 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:12:27.218065 6155 external_mini_cluster.cc:1428] Setting key 987a9921221e09a5c8b4b5e803e19627
I20260924 10:12:27.220779 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36177
--builtin_ntp_servers=127.6.2.212:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:12:27.227777 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.216106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:47199 (local address 127.6.2.254:36177)
0924 10:12:27.216292 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:27.216297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:27.217812 (+ 1515us) server_negotiation.cc:407] Connection header received
0924 10:12:27.219970 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:27.219975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:27.220059 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:27.220179 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:27.221877 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.222721 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.224765 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.225204 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.225376 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.226945 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:27.226950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:27.226953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:27.226971 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:27.227004 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:27.227017 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:27.227185 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:27.227446 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 10:12:27.227536 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:12:27.229355 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.216575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36177 (local address 127.6.2.194:47199)
0924 10:12:27.217608 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:27.217692 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:27.219675 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:27.220413 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:27.220422 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:27.220509 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:27.221564 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.221601 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.222926 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.222932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.224537 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.224559 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.225471 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.225478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.225771 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.226553 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:12:27.226556 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:27.226596 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:12:27.226687 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:27.227190 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:27.227204 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:27.227214 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:12:27.227429 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 10:12:27.227918 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":374,"threads_started":1}
I20260924 10:12:27.230829 24204 heartbeater.cc:347] Connected to a master server at 127.6.2.254:36177
I20260924 10:12:27.231174 24204 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:27.231937 24204 heartbeater.cc:511] Master 127.6.2.254:36177 requested a full tablet report, sending...
I20260924 10:12:27.233420 23887 ts_manager.cc:194] Registered new tserver with Master: 1fcf43e0c68b4067bda473f44be13699 (127.6.2.194:41567)
I20260924 10:12:27.234232 23887 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.2.194:47199
W20260924 10:12:27.548012 24207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:27.548308 24207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:27.548477 24207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:27.555902 24207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:27.556007 24207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:27.556169 24207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:12:27.565579 24207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:36177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:27.566903 24207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:27.568702 24207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:27.581741 24212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:27.581864 24215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:27.582055 24213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:27.583128 24207 server_base.cc:1034] running on GCE node
I20260924 10:12:27.583789 24207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:27.584862 24207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:27.586095 24207 hybrid_clock.cc:648] HybridClock initialized: now 1790244747586039 us; error 48 us; skew 500 ppm
I20260924 10:12:27.586439 24207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:12:27.588750 24207 webserver.cc:460] Webserver started at http://127.6.2.195:37185/ using document root <none> and password file <none>
I20260924 10:12:27.589444 24207 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:27.589529 24207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:27.589843 24207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:27.592759 24207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "077689d14a054a79a3f3126a626ffd8d"
format_stamp: "Formatted at 2026-09-24 10:12:27 on dist-test-slave-s3n3"
server_key: "6ac25a05bf7a2a5d0cf6621ff2a3692b"
server_key_iv: "911cfce1bb38fe72013105aa5cfed5b0"
server_key_version: "encryptionkey@0"
I20260924 10:12:27.593557 24207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "077689d14a054a79a3f3126a626ffd8d"
format_stamp: "Formatted at 2026-09-24 10:12:27 on dist-test-slave-s3n3"
server_key: "6ac25a05bf7a2a5d0cf6621ff2a3692b"
server_key_iv: "911cfce1bb38fe72013105aa5cfed5b0"
server_key_version: "encryptionkey@0"
I20260924 10:12:27.598603 24207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 10:12:27.602000 24221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:27.603319 24207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:12:27.603502 24207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "077689d14a054a79a3f3126a626ffd8d"
format_stamp: "Formatted at 2026-09-24 10:12:27 on dist-test-slave-s3n3"
server_key: "6ac25a05bf7a2a5d0cf6621ff2a3692b"
server_key_iv: "911cfce1bb38fe72013105aa5cfed5b0"
server_key_version: "encryptionkey@0"
I20260924 10:12:27.603718 24207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:27.623271 24207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:27.624135 24207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:27.624426 24207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:27.625451 24207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:27.626924 24207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:27.627054 24207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:27.627229 24207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:27.627316 24207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:27.659983 24207 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:40373
I20260924 10:12:27.659997 24333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:40373 every 8 connection(s)
I20260924 10:12:27.661273 24207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:12:27.666411 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 24207
I20260924 10:12:27.666602 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:12:27.667022 6155 external_mini_cluster.cc:1428] Setting key 40e8702f9550007726dc4835d8894301
I20260924 10:12:27.674258 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.663946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:41643 (local address 127.6.2.254:36177)
0924 10:12:27.664124 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:27.664129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:27.665539 (+ 1410us) server_negotiation.cc:407] Connection header received
0924 10:12:27.667527 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:27.667534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:27.667631 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:27.667745 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:27.669243 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.670233 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.671824 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.672277 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.672373 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.673612 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:27.673617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:27.673620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:27.673638 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:27.673670 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:27.673681 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:27.673751 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:27.673945 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 10:12:27.674019 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:12:27.675295 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.664442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:36177 (local address 127.6.2.195:41643)
0924 10:12:27.665365 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:27.665401 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:27.667223 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:27.668064 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:27.668082 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:27.668172 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:27.669072 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.669096 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.670378 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.670389 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.671703 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.671715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.672470 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.672475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.672666 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.673305 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:12:27.673308 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:27.673345 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:12:27.673439 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:27.673763 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:27.673770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:27.673776 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:12:27.673914 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 10:12:27.674204 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":319,"threads_started":1}
I20260924 10:12:27.676532 24334 heartbeater.cc:347] Connected to a master server at 127.6.2.254:36177
I20260924 10:12:27.676827 24334 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:27.677564 24334 heartbeater.cc:511] Master 127.6.2.254:36177 requested a full tablet report, sending...
I20260924 10:12:27.678889 23887 ts_manager.cc:194] Registered new tserver with Master: 077689d14a054a79a3f3126a626ffd8d (127.6.2.195:40373)
I20260924 10:12:27.679787 23887 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.2.195:41643
I20260924 10:12:27.682428 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:12:27.692639 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.686764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.6.2.254:36177)
0924 10:12:27.687045 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:27.687050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:27.687353 (+ 303us) server_negotiation.cc:407] Connection header received
0924 10:12:27.687495 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:27.687499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:27.687571 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:27.687653 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:27.688132 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.688808 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.690202 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.690588 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.690678 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.691940 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:27.691944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:27.691948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:27.691965 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:27.692004 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:27.692017 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:27.692089 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:27.692211 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 10:12:27.692289 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:12:27.696594 23887 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:12:27.704222 23887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:12:27.729562 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.717525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.6.2.193:43405)
0924 10:12:27.719027 (+ 1502us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:27.719039 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:27.719119 (+ 80us) server_negotiation.cc:407] Connection header received
0924 10:12:27.719252 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:27.719263 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:27.720132 (+ 869us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:27.721165 (+ 1033us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:27.721462 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.723562 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.726682 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.728066 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.728197 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.728444 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:27.729018 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 10:12:27.729197 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1295,"thread_start_us":1180,"threads_started":1}
I20260924 10:12:27.729561 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.717464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:43405 (local address 127.0.0.1:33362)
0924 10:12:27.718210 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:27.718255 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:27.718427 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:27.720376 (+ 1949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:27.720402 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:27.720445 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:27.720918 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.720936 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.723761 (+ 2825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.723770 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.726491 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.726540 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.728559 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.728566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.728760 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.728808 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 10:12:27.728942 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 10:12:27.729119 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":139,"thread_start_us":353,"threads_started":1}
I20260924 10:12:27.730813 24348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.718642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:40373 (local address 127.0.0.1:48634)
0924 10:12:27.720028 (+ 1386us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:27.720045 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:27.720165 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:27.721152 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:27.721159 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:27.721195 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:27.721727 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.721741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.724033 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.724037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.727851 (+ 3814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.727868 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.729948 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.729955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.730196 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.730234 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 10:12:27.730330 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 10:12:27.730458 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1255,"mutex_wait_us":60,"thread_start_us":1098,"threads_started":1}
I20260924 10:12:27.731690 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.719013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48634 (local address 127.6.2.195:40373)
0924 10:12:27.719782 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:27.719791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:27.720294 (+ 503us) server_negotiation.cc:407] Connection header received
0924 10:12:27.720414 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:27.720421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:27.720633 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:27.720818 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:27.721990 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.723874 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.728097 (+ 4223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.729396 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.729531 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.729800 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:27.731001 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0924 10:12:27.731362 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":451,"threads_started":1}
I20260924 10:12:27.733744 24008 tablet_service.cc:1467] Processing CreateTablet for tablet b9f4430b33ea4cd2a0bfe83a69b747c4 (DEFAULT_TABLE table=test-table [id=20dbd01dafe74981b5cd39a68dfd6913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:27.734369 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.720254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:41567 (local address 127.0.0.1:54336)
0924 10:12:27.726440 (+ 6186us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:27.726457 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:27.726554 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:27.727179 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:27.727184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:27.727211 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:27.727594 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.727605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.729372 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.729376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.731445 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.731459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.733636 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.733642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.733822 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.733858 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:12:27.733966 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 10:12:27.734090 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6067,"thread_start_us":5926,"threads_started":1}
I20260924 10:12:27.734512 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.720459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.6.2.194:41567)
0924 10:12:27.725993 (+ 5534us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:27.726005 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:27.726566 (+ 561us) server_negotiation.cc:407] Connection header received
0924 10:12:27.726699 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:27.726709 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:27.726951 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:27.727173 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:27.727777 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.729196 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.731648 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.733279 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.733431 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.733680 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:27.734012 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 10:12:27.734135 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5326,"thread_start_us":5186,"threads_started":1}
I20260924 10:12:27.736256 24008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f4430b33ea4cd2a0bfe83a69b747c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:27.736929 24269 tablet_service.cc:1467] Processing CreateTablet for tablet b9f4430b33ea4cd2a0bfe83a69b747c4 (DEFAULT_TABLE table=test-table [id=20dbd01dafe74981b5cd39a68dfd6913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:27.738075 24269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f4430b33ea4cd2a0bfe83a69b747c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:27.742302 24139 tablet_service.cc:1467] Processing CreateTablet for tablet b9f4430b33ea4cd2a0bfe83a69b747c4 (DEFAULT_TABLE table=test-table [id=20dbd01dafe74981b5cd39a68dfd6913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:27.743595 24139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f4430b33ea4cd2a0bfe83a69b747c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:27.750396 24352 tablet_bootstrap.cc:492] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c: Bootstrap starting.
I20260924 10:12:27.753525 24352 tablet_bootstrap.cc:654] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:27.754508 24352 log.cc:824] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:27.757793 24354 tablet_bootstrap.cc:492] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699: Bootstrap starting.
I20260924 10:12:27.758620 24352 tablet_bootstrap.cc:492] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c: No bootstrap required, opened a new log
I20260924 10:12:27.758965 24352 ts_tablet_manager.cc:1397] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 10:12:27.759330 24355 tablet_bootstrap.cc:492] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d: Bootstrap starting.
I20260924 10:12:27.761343 24354 tablet_bootstrap.cc:654] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:27.761989 24355 tablet_bootstrap.cc:654] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:27.762506 24354 log.cc:824] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:27.762513 24073 heartbeater.cc:503] Master 127.6.2.254:36177 was elected leader, sending a full tablet report...
I20260924 10:12:27.763161 24355 log.cc:824] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:27.770609 24355 tablet_bootstrap.cc:492] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d: No bootstrap required, opened a new log
I20260924 10:12:27.770931 24355 ts_tablet_manager.cc:1397] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260924 10:12:27.771391 24354 tablet_bootstrap.cc:492] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699: No bootstrap required, opened a new log
I20260924 10:12:27.771665 24354 ts_tablet_manager.cc:1397] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20260924 10:12:27.771687 24352 raft_consensus.cc:348] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dd373e9f63484e8001c97e412b427c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43405 } } peers { permanent_uuid: "077689d14a054a79a3f3126a626ffd8d" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 40373 } } peers { permanent_uuid: "1fcf43e0c68b4067bda473f44be13699" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41567 } }
I20260924 10:12:27.772145 24352 raft_consensus.cc:374] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:27.772281 24352 raft_consensus.cc:729] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9dd373e9f63484e8001c97e412b427c, State: Initialized, Role: FOLLOWER
I20260924 10:12:27.772904 24352 consensus_queue.cc:260] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dd373e9f63484e8001c97e412b427c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43405 } } peers { permanent_uuid: "077689d14a054a79a3f3126a626ffd8d" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 40373 } } peers { permanent_uuid: "1fcf43e0c68b4067bda473f44be13699" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41567 } }
I20260924 10:12:27.774685 24352 ts_tablet_manager.cc:1428] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20260924 10:12:27.780701 24355 raft_consensus.cc:348] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dd373e9f63484e8001c97e412b427c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43405 } } peers { permanent_uuid: "077689d14a054a79a3f3126a626ffd8d" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 40373 } } peers { permanent_uuid: "1fcf43e0c68b4067bda473f44be13699" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41567 } }
I20260924 10:12:27.781070 24355 raft_consensus.cc:374] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:27.781193 24355 raft_consensus.cc:729] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077689d14a054a79a3f3126a626ffd8d, State: Initialized, Role: FOLLOWER
I20260924 10:12:27.780957 24354 raft_consensus.cc:348] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dd373e9f63484e8001c97e412b427c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43405 } } peers { permanent_uuid: "077689d14a054a79a3f3126a626ffd8d" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 40373 } } peers { permanent_uuid: "1fcf43e0c68b4067bda473f44be13699" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41567 } }
I20260924 10:12:27.781451 24354 raft_consensus.cc:374] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:27.781584 24354 raft_consensus.cc:729] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcf43e0c68b4067bda473f44be13699, State: Initialized, Role: FOLLOWER
I20260924 10:12:27.781896 24355 consensus_queue.cc:260] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dd373e9f63484e8001c97e412b427c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43405 } } peers { permanent_uuid: "077689d14a054a79a3f3126a626ffd8d" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 40373 } } peers { permanent_uuid: "1fcf43e0c68b4067bda473f44be13699" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41567 } }
I20260924 10:12:27.782081 24354 consensus_queue.cc:260] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dd373e9f63484e8001c97e412b427c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43405 } } peers { permanent_uuid: "077689d14a054a79a3f3126a626ffd8d" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 40373 } } peers { permanent_uuid: "1fcf43e0c68b4067bda473f44be13699" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41567 } }
I20260924 10:12:27.783121 24334 heartbeater.cc:503] Master 127.6.2.254:36177 was elected leader, sending a full tablet report...
I20260924 10:12:27.783185 24204 heartbeater.cc:503] Master 127.6.2.254:36177 was elected leader, sending a full tablet report...
I20260924 10:12:27.783932 24354 ts_tablet_manager.cc:1428] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260924 10:12:27.783900 24355 ts_tablet_manager.cc:1428] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260924 10:12:27.816283 24358 raft_consensus.cc:482] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:12:27.816634 24358 raft_consensus.cc:504] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dd373e9f63484e8001c97e412b427c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43405 } } peers { permanent_uuid: "077689d14a054a79a3f3126a626ffd8d" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 40373 } } peers { permanent_uuid: "1fcf43e0c68b4067bda473f44be13699" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41567 } }
I20260924 10:12:27.818358 24358 leader_election.cc:288] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 077689d14a054a79a3f3126a626ffd8d (127.6.2.195:40373), 1fcf43e0c68b4067bda473f44be13699 (127.6.2.194:41567)
I20260924 10:12:27.824787 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.818883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:56615 (local address 127.6.2.195:40373)
0924 10:12:27.819174 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:27.819183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:27.819522 (+ 339us) server_negotiation.cc:407] Connection header received
0924 10:12:27.819768 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:27.819775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:27.819878 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:27.819976 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:27.820582 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.821315 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.823114 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.823888 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.824063 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.824163 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:27.824418 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 10:12:27.824541 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:12:27.824860 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.818728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:40373 (local address 127.6.2.193:56615)
0924 10:12:27.819375 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:27.819392 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:27.819584 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:27.820057 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:27.820061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:27.820092 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:27.820442 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.820450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.821458 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.821465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.822904 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.822920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.824076 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.824082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.824254 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.824282 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:12:27.824391 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 10:12:27.824523 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":328,"threads_started":1}
I20260924 10:12:27.825287 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.819771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:50891 (local address 127.6.2.194:41567)
0924 10:12:27.819985 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:27.819992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:27.820263 (+ 271us) server_negotiation.cc:407] Connection header received
0924 10:12:27.820477 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:27.820486 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:27.820555 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:27.820658 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:27.821159 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.821821 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.823588 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.824388 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.824528 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.824626 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:27.824914 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 10:12:27.825036 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 10:12:27.825963 24289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f4430b33ea4cd2a0bfe83a69b747c4" candidate_uuid: "b9dd373e9f63484e8001c97e412b427c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077689d14a054a79a3f3126a626ffd8d" is_pre_election: true
I20260924 10:12:27.826362 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.819520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:41567 (local address 127.6.2.193:50891)
0924 10:12:27.820155 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:27.820172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:27.820327 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:27.820674 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:27.820678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:27.820699 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:27.821013 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.821022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.821969 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.821978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.823413 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:27.823425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.824590 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.824596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:27.824749 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.824772 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:12:27.824890 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 10:12:27.825324 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":308,"threads_started":1}
I20260924 10:12:27.826524 24289 raft_consensus.cc:2454] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9dd373e9f63484e8001c97e412b427c in term 0.
I20260924 10:12:27.827298 23963 leader_election.cc:302] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077689d14a054a79a3f3126a626ffd8d, b9dd373e9f63484e8001c97e412b427c; no voters:
I20260924 10:12:27.827505 24159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f4430b33ea4cd2a0bfe83a69b747c4" candidate_uuid: "b9dd373e9f63484e8001c97e412b427c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf43e0c68b4067bda473f44be13699" is_pre_election: true
I20260924 10:12:27.827728 24358 raft_consensus.cc:2781] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:12:27.827950 24358 raft_consensus.cc:482] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:12:27.828056 24358 raft_consensus.cc:3037] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:27.828060 24159 raft_consensus.cc:2454] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9dd373e9f63484e8001c97e412b427c in term 0.
I20260924 10:12:27.830386 24358 raft_consensus.cc:504] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dd373e9f63484e8001c97e412b427c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43405 } } peers { permanent_uuid: "077689d14a054a79a3f3126a626ffd8d" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 40373 } } peers { permanent_uuid: "1fcf43e0c68b4067bda473f44be13699" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41567 } }
I20260924 10:12:27.831014 24358 leader_election.cc:288] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [CANDIDATE]: Term 1 election: Requested vote from peers 077689d14a054a79a3f3126a626ffd8d (127.6.2.195:40373), 1fcf43e0c68b4067bda473f44be13699 (127.6.2.194:41567)
I20260924 10:12:27.831588 24289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f4430b33ea4cd2a0bfe83a69b747c4" candidate_uuid: "b9dd373e9f63484e8001c97e412b427c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077689d14a054a79a3f3126a626ffd8d"
I20260924 10:12:27.831765 24159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f4430b33ea4cd2a0bfe83a69b747c4" candidate_uuid: "b9dd373e9f63484e8001c97e412b427c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcf43e0c68b4067bda473f44be13699"
I20260924 10:12:27.831825 24289 raft_consensus.cc:3037] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:27.831998 24159 raft_consensus.cc:3037] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:27.834205 24289 raft_consensus.cc:2454] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9dd373e9f63484e8001c97e412b427c in term 1.
I20260924 10:12:27.834221 24159 raft_consensus.cc:2454] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9dd373e9f63484e8001c97e412b427c in term 1.
I20260924 10:12:27.834774 23963 leader_election.cc:302] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcf43e0c68b4067bda473f44be13699, b9dd373e9f63484e8001c97e412b427c; no voters:
I20260924 10:12:27.835127 24358 raft_consensus.cc:2781] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:27.835604 24358 raft_consensus.cc:686] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [term 1 LEADER]: Becoming Leader. State: Replica: b9dd373e9f63484e8001c97e412b427c, State: Running, Role: LEADER
I20260924 10:12:27.836105 24358 consensus_queue.cc:237] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dd373e9f63484e8001c97e412b427c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43405 } } peers { permanent_uuid: "077689d14a054a79a3f3126a626ffd8d" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 40373 } } peers { permanent_uuid: "1fcf43e0c68b4067bda473f44be13699" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41567 } }
I20260924 10:12:27.841173 23887 catalog_manager.cc:5495] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c reported cstate change: term changed from 0 to 1, leader changed from <none> to b9dd373e9f63484e8001c97e412b427c (127.6.2.193). New cstate: current_term: 1 leader_uuid: "b9dd373e9f63484e8001c97e412b427c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dd373e9f63484e8001c97e412b427c" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077689d14a054a79a3f3126a626ffd8d" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 40373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcf43e0c68b4067bda473f44be13699" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41567 } health_report { overall_health: UNKNOWN } } }
I20260924 10:12:27.882978 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.877568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.6.2.193:43405)
0924 10:12:27.877798 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:27.877805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:27.877825 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:12:27.878026 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:27.878030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:27.878127 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:27.878284 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:12:27.878652 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.879415 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.880783 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.881208 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.881343 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.881813 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:12:27.882340 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:12:27.882499 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:27.882610 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 10:12:27.882693 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 10:12:27.891710 24289 raft_consensus.cc:1261] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d [term 1 FOLLOWER]: Refusing update from remote peer b9dd373e9f63484e8001c97e412b427c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:12:27.891710 24159 raft_consensus.cc:1261] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699 [term 1 FOLLOWER]: Refusing update from remote peer b9dd373e9f63484e8001c97e412b427c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:12:27.892654 24363 consensus_queue.cc:1035] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [LEADER]: Connected to new peer: Peer: permanent_uuid: "077689d14a054a79a3f3126a626ffd8d" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:12:27.893069 24358 consensus_queue.cc:1035] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcf43e0c68b4067bda473f44be13699" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:12:27.904253 24366 mvcc.cc:205] Tried to move back new op lower bound from 7332842487353909248 to 7332842487140376576. Current Snapshot: MvccSnapshot[applied={T|T < 7332842487353909248}]
I20260924 10:12:27.919210 23887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:12:27.919479 23887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:12:27.923296 23887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2fdccfd15f94fa88b196ed4edb1db21: Sending DeleteTablet for 3 replicas of tablet b9f4430b33ea4cd2a0bfe83a69b747c4
I20260924 10:12:27.924535 24008 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f4430b33ea4cd2a0bfe83a69b747c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:27 UTC) from {username='slave'} at 127.0.0.1:33362
I20260924 10:12:27.924666 24139 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f4430b33ea4cd2a0bfe83a69b747c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:27 UTC) from {username='slave'} at 127.0.0.1:54336
I20260924 10:12:27.924722 24269 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f4430b33ea4cd2a0bfe83a69b747c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:27 UTC) from {username='slave'} at 127.0.0.1:48634
I20260924 10:12:27.927446 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 23946
I20260924 10:12:27.937176 24376 tablet_replica.cc:330] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c: stopping tablet replica
I20260924 10:12:27.938052 24376 raft_consensus.cc:2229] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:12:27.939077 24376 raft_consensus.cc:2258] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:27.942360 24376 ts_tablet_manager.cc:1905] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:27.943127 24377 tablet_replica.cc:330] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699: stopping tablet replica
I20260924 10:12:27.943653 24377 raft_consensus.cc:2229] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:12:27.944300 24377 raft_consensus.cc:2258] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:27.950085 24378 tablet_replica.cc:330] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d: stopping tablet replica
I20260924 10:12:27.951022 24378 raft_consensus.cc:2229] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:12:27.951673 24378 raft_consensus.cc:2258] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:27.951802 24377 ts_tablet_manager.cc:1905] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:27.952579 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:27.929987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.6.2.193:43405)
0924 10:12:27.930440 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:27.930446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:27.943452 (+ 13006us) server_negotiation.cc:407] Connection header received
0924 10:12:27.943771 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:27.943779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:27.943893 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:27.943982 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:27.944703 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.945930 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.947951 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:27.948600 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:27.948699 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:27.950690 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:27.950701 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:27.950720 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:27.951567 (+ 847us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:27.951641 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:27.951667 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:27.951778 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:27.951998 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 10:12:27.952111 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260924 10:12:27.954205 24068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:27.957392 24378 ts_tablet_manager.cc:1905] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:27.959506 24377 ts_tablet_manager.cc:1918] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:27.959830 24377 log.cc:1196] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/b9f4430b33ea4cd2a0bfe83a69b747c4
I20260924 10:12:27.960393 24377 ts_tablet_manager.cc:1939] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 1fcf43e0c68b4067bda473f44be13699: Deleting consensus metadata
I20260924 10:12:27.962476 23873 catalog_manager.cc:4841] TS 1fcf43e0c68b4067bda473f44be13699 (127.6.2.194:41567): tablet b9f4430b33ea4cd2a0bfe83a69b747c4 (table test-table [id=20dbd01dafe74981b5cd39a68dfd6913]) successfully deleted
I20260924 10:12:27.964483 24378 ts_tablet_manager.cc:1918] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:27.964648 24378 log.cc:1196] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/b9f4430b33ea4cd2a0bfe83a69b747c4
I20260924 10:12:27.965070 24378 ts_tablet_manager.cc:1939] T b9f4430b33ea4cd2a0bfe83a69b747c4 P 077689d14a054a79a3f3126a626ffd8d: Deleting consensus metadata
I20260924 10:12:27.966892 23873 catalog_manager.cc:4841] TS 077689d14a054a79a3f3126a626ffd8d (127.6.2.195:40373): tablet b9f4430b33ea4cd2a0bfe83a69b747c4 (table test-table [id=20dbd01dafe74981b5cd39a68dfd6913]) successfully deleted
I20260924 10:12:28.054838 24376 ts_tablet_manager.cc:1918] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:28.055186 24376 log.cc:1196] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/b9f4430b33ea4cd2a0bfe83a69b747c4
I20260924 10:12:28.055776 24376 ts_tablet_manager.cc:1939] T b9f4430b33ea4cd2a0bfe83a69b747c4 P b9dd373e9f63484e8001c97e412b427c: Deleting consensus metadata
I20260924 10:12:28.056595 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 23946
I20260924 10:12:28.079298 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 24077
W20260924 10:12:28.079288 23873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.2.193:43405 (error 108)
I20260924 10:12:28.081162 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:28.080729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:43405 (local address 127.0.0.1:33394)
0924 10:12:28.080915 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:28.081025 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:43405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 10:12:28.081482 23873 catalog_manager.cc:4584] TS b9dd373e9f63484e8001c97e412b427c (127.6.2.193:43405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9f4430b33ea4cd2a0bfe83a69b747c4: Network error: Client connection negotiation failed: client connection to 127.6.2.193:43405: connect: Connection refused (error 111)
I20260924 10:12:28.086488 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:28.080195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.6.2.194:41567)
0924 10:12:28.080378 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:28.080384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:28.080459 (+ 75us) server_negotiation.cc:407] Connection header received
0924 10:12:28.080679 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:28.080684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:28.080821 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:28.080951 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:28.081339 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:28.082157 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:28.083549 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:28.083958 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:28.084089 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:28.085137 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:28.085146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:28.085159 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:28.085727 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:28.085785 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:28.085804 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:28.085942 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:28.086060 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 10:12:28.086167 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:12:28.087208 24199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:28.135370 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:28.134828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:43405 (local address 127.0.0.1:33398)
0924 10:12:28.135117 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:28.135205 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:43405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 10:12:28.136335 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:28.136009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:43405 (local address 127.0.0.1:33412)
0924 10:12:28.136163 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:28.136227 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:43405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 10:12:28.180480 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 24077
I20260924 10:12:28.196907 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:28.196261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:43405 (local address 127.0.0.1:33428)
0924 10:12:28.196538 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:28.196664 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:43405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 10:12:28.198153 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:28.197705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:43405 (local address 127.0.0.1:33432)
0924 10:12:28.197894 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:28.197991 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:43405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 10:12:28.200229 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 24207
I20260924 10:12:28.207194 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:28.201024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.6.2.195:40373)
0924 10:12:28.201226 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:28.201233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:28.201254 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:12:28.201452 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:28.201457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:28.201542 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:28.201656 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:28.202070 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:28.202801 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:28.204126 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:28.204523 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:28.204662 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:28.205754 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:28.205765 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:28.205781 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:28.206358 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:28.206406 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:28.206426 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:28.206563 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:28.206721 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 10:12:28.206834 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:12:28.207904 24329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:28.278483 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:28.277974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:43405 (local address 127.0.0.1:33440)
0924 10:12:28.278217 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:28.278319 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:43405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 10:12:28.279601 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:28.279224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:43405 (local address 127.0.0.1:33448)
0924 10:12:28.279421 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:28.279503 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:43405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 10:12:28.299431 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 24207
I20260924 10:12:28.316669 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 23856
I20260924 10:12:28.317883 23917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:28.427243 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 23856
2026-09-24T10:12:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2781 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:12:28 dist-test-slave-s3n3 krb5kdc[24387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:12:28 dist-test-slave-s3n3 krb5kdc[24387](info): set up 2 sockets
Sep 24 10:12:28 dist-test-slave-s3n3 krb5kdc[24387](info): commencing operation
krb5kdc: starting...
W20260924 10:12:29.551565 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:12:29 dist-test-slave-s3n3 krb5kdc[24387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:12:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:12:29.819439 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:44193
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:44193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:30.151710 24400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:30.152026 24400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:30.152117 24400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:30.159602 24400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:30.159758 24400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:30.159863 24400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:30.159953 24400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:30.160038 24400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:30.170339 24400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:44193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:44193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:30.171823 24400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:30.173795 24400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:30.187371 24405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:30.187283 24406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:30.187312 24408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:30.188019 24400 server_base.cc:1034] running on GCE node
I20260924 10:12:30.188824 24400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:30.190395 24400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:30.191620 24400 hybrid_clock.cc:648] HybridClock initialized: now 1790244750191574 us; error 40 us; skew 500 ppm
I20260924 10:12:30.191977 24400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:30 dist-test-slave-s3n3 krb5kdc[24387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244750, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:30.195487 24400 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:12:30.197080 24400 webserver.cc:460] Webserver started at http://127.6.2.254:37955/ using document root <none> and password file <none>
I20260924 10:12:30.197768 24400 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:30.197881 24400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:30.198164 24400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:30.201179 24400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "b233dcb2985b42f2b41493f9fd06036e"
format_stamp: "Formatted at 2026-09-24 10:12:30 on dist-test-slave-s3n3"
server_key: "eda86bcb3e5f3a38b6e9bfa836cc4cfc"
server_key_iv: "4000b8b3bc14a7ccfba19baec834161e"
server_key_version: "encryptionkey@0"
I20260924 10:12:30.201900 24400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "b233dcb2985b42f2b41493f9fd06036e"
format_stamp: "Formatted at 2026-09-24 10:12:30 on dist-test-slave-s3n3"
server_key: "eda86bcb3e5f3a38b6e9bfa836cc4cfc"
server_key_iv: "4000b8b3bc14a7ccfba19baec834161e"
server_key_version: "encryptionkey@0"
I20260924 10:12:30.207217 24400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:12:30.210706 24415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:30.212083 24400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:12:30.212265 24400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "b233dcb2985b42f2b41493f9fd06036e"
format_stamp: "Formatted at 2026-09-24 10:12:30 on dist-test-slave-s3n3"
server_key: "eda86bcb3e5f3a38b6e9bfa836cc4cfc"
server_key_iv: "4000b8b3bc14a7ccfba19baec834161e"
server_key_version: "encryptionkey@0"
I20260924 10:12:30.212474 24400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:30.229025 24400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:30.234431 24400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:30.234754 24400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:30.254685 24400 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:44193
I20260924 10:12:30.254724 24466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:44193 every 8 connection(s)
I20260924 10:12:30.256237 24400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:30.260048 24467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:30.264720 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 24400
I20260924 10:12:30.264937 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:30.265543 6155 external_mini_cluster.cc:1428] Setting key c78241e1147510129cc395821ce666d6
I20260924 10:12:30.271895 24467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e: Bootstrap starting.
I20260924 10:12:30.275012 24467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:30.276099 24467 log.cc:824] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:30.278913 24467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e: No bootstrap required, opened a new log
Sep 24 10:12:30 dist-test-slave-s3n3 krb5kdc[24387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:12:30.287038 24467 raft_consensus.cc:348] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b233dcb2985b42f2b41493f9fd06036e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44193 } }
I20260924 10:12:30.287516 24467 raft_consensus.cc:374] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:30.287629 24467 raft_consensus.cc:729] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b233dcb2985b42f2b41493f9fd06036e, State: Initialized, Role: FOLLOWER
I20260924 10:12:30.288075 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:30.267597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.6.2.254:44193)
0924 10:12:30.268462 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:30.268477 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:30.268522 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:12:30.270492 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:30.270514 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:30.270900 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:30.271336 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:30.272829 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:30.274158 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:30.275619 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:30.276337 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:30.276518 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:30.280537 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:30.280548 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:30.280563 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:30.280602 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:30.283836 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:30.284571 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:30.284578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:30.284587 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:30.284756 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:30.285162 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:30.285165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:30.285167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:30.285720 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:30.286044 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:30.286487 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 10:12:30.286905 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":366,"threads_started":1}
I20260924 10:12:30.288348 24467 consensus_queue.cc:260] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b233dcb2985b42f2b41493f9fd06036e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44193 } }
I20260924 10:12:30.288652 24467 raft_consensus.cc:388] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:30.288831 24467 raft_consensus.cc:482] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:30.289031 24467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:30.291702 24467 raft_consensus.cc:504] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b233dcb2985b42f2b41493f9fd06036e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44193 } }
I20260924 10:12:30.292145 24467 leader_election.cc:302] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b233dcb2985b42f2b41493f9fd06036e; no voters:
I20260924 10:12:30.292732 24467 leader_election.cc:288] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:30.292933 24472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:30.293679 24472 raft_consensus.cc:686] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [term 1 LEADER]: Becoming Leader. State: Replica: b233dcb2985b42f2b41493f9fd06036e, State: Running, Role: LEADER
I20260924 10:12:30.294199 24472 consensus_queue.cc:237] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b233dcb2985b42f2b41493f9fd06036e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44193 } }
I20260924 10:12:30.294822 24467 sys_catalog.cc:564] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:30.298163 24473 sys_catalog.cc:455] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b233dcb2985b42f2b41493f9fd06036e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b233dcb2985b42f2b41493f9fd06036e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44193 } } }
I20260924 10:12:30.298478 24474 sys_catalog.cc:455] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b233dcb2985b42f2b41493f9fd06036e. Latest consensus state: current_term: 1 leader_uuid: "b233dcb2985b42f2b41493f9fd06036e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b233dcb2985b42f2b41493f9fd06036e" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 44193 } } }
I20260924 10:12:30.298797 24473 sys_catalog.cc:458] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:30.298854 24474 sys_catalog.cc:458] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:30.299865 24479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:30.305629 24479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:30.315339 24479 catalog_manager.cc:1347] Generated new cluster ID: 99a2a8cd07604170b719417f640b95de
I20260924 10:12:30.315469 24479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:30.332405 24479 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:30.334496 24479 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:30.361581 24479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e: Generated new TSK 0
I20260924 10:12:30.362578 24479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:12:30.432881 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:44193
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:30.774514 24493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:30.774885 24493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:30.775090 24493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:30.782522 24493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:30.782684 24493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:30.782876 24493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:12:30.793324 24493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:44193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:30.794744 24493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:30.796162 24493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:30.810185 24498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:30.810140 24499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:30.810148 24501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:30.811211 24493 server_base.cc:1034] running on GCE node
I20260924 10:12:30.811883 24493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:30.812915 24493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:30.814136 24493 hybrid_clock.cc:648] HybridClock initialized: now 1790244750814101 us; error 34 us; skew 500 ppm
I20260924 10:12:30.814504 24493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:30 dist-test-slave-s3n3 krb5kdc[24387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244750, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:30.817999 24493 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:12:30.819531 24493 webserver.cc:460] Webserver started at http://127.6.2.193:46625/ using document root <none> and password file <none>
I20260924 10:12:30.820199 24493 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:30.820294 24493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:30.820557 24493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:30.823469 24493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "f383722ea9f242428151cb9ec237f283"
format_stamp: "Formatted at 2026-09-24 10:12:30 on dist-test-slave-s3n3"
server_key: "3fbdca72b2599adc44554014b316240a"
server_key_iv: "7d62708bc2e8c10aa7ef0fab6d35eb3d"
server_key_version: "encryptionkey@0"
I20260924 10:12:30.824198 24493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "f383722ea9f242428151cb9ec237f283"
format_stamp: "Formatted at 2026-09-24 10:12:30 on dist-test-slave-s3n3"
server_key: "3fbdca72b2599adc44554014b316240a"
server_key_iv: "7d62708bc2e8c10aa7ef0fab6d35eb3d"
server_key_version: "encryptionkey@0"
I20260924 10:12:30.829298 24493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:12:30.833029 24508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:30.834352 24493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:12:30.834547 24493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "f383722ea9f242428151cb9ec237f283"
format_stamp: "Formatted at 2026-09-24 10:12:30 on dist-test-slave-s3n3"
server_key: "3fbdca72b2599adc44554014b316240a"
server_key_iv: "7d62708bc2e8c10aa7ef0fab6d35eb3d"
server_key_version: "encryptionkey@0"
I20260924 10:12:30.834761 24493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:30.863175 24493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:30.867893 24493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:30.868214 24493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:30.869228 24493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:30.870687 24493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:30.870790 24493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:30.870944 24493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:30.871018 24493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:30.906270 24493 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:40303
I20260924 10:12:30.906365 24620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:40303 every 8 connection(s)
I20260924 10:12:30.907703 24493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:12:30.911909 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 24493
I20260924 10:12:30.912117 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:12:30.912585 6155 external_mini_cluster.cc:1428] Setting key 1597e0589873b0f66e7f6a3e993c0e20
Sep 24 10:12:30 dist-test-slave-s3n3 krb5kdc[24387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244750, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:12:30.928751 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:30.910360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:54805 (local address 127.6.2.254:44193)
0924 10:12:30.910855 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:30.910859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:30.911680 (+ 821us) server_negotiation.cc:407] Connection header received
0924 10:12:30.913893 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:30.913899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:30.913992 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:30.914114 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:30.915694 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:30.916589 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:30.918593 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:30.919020 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:30.919208 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:30.922789 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:30.922794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:30.922797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:30.922831 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:30.925491 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:30.926284 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:30.926290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:30.926294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:30.926424 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:30.926870 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:30.926877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:30.926882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:30.927246 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:30.927403 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:30.928124 (+ 721us) server_negotiation.cc:299] Negotiation successful
0924 10:12:30.928378 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":287,"threads_started":1}
I20260924 10:12:30.929780 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:30.910754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:44193 (local address 127.6.2.193:54805)
0924 10:12:30.911520 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:30.911557 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:30.913599 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:30.914362 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:30.914370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:30.914890 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:30.915521 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:30.915540 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:30.916772 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:30.916776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:30.918433 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:30.918447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:30.919296 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:30.919305 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:30.919556 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:30.920285 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:12:30.920308 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:30.922524 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:30.925653 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:30.925663 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:30.925691 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:30.926151 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:30.926539 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:30.926544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:30.926548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:30.926753 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:30.927375 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:30.927384 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:30.927744 (+ 360us) client_negotiation.cc:769] Sending connection context
0924 10:12:30.928028 (+ 284us) client_negotiation.cc:240] Negotiation successful
0924 10:12:30.928584 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":343,"threads_started":1}
I20260924 10:12:30.931615 24621 heartbeater.cc:347] Connected to a master server at 127.6.2.254:44193
I20260924 10:12:30.931977 24621 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:30.932917 24621 heartbeater.cc:511] Master 127.6.2.254:44193 requested a full tablet report, sending...
I20260924 10:12:30.935724 24432 ts_manager.cc:194] Registered new tserver with Master: f383722ea9f242428151cb9ec237f283 (127.6.2.193:40303)
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
I20260924 10:12:30.938062 24432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:54805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:12:30.971566 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:44193
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:31.312491 24627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:31.312844 24627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:31.313019 24627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:31.320829 24627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:31.320998 24627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:31.321173 24627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:12:31.331910 24627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:44193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:31.333269 24627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:31.334599 24627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:31.348341 24632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:31.348384 24635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:31.348414 24633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:31.350124 24627 server_base.cc:1034] running on GCE node
I20260924 10:12:31.351279 24627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:31.352831 24627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:31.354212 24627 hybrid_clock.cc:648] HybridClock initialized: now 1790244751354157 us; error 55 us; skew 500 ppm
I20260924 10:12:31.354771 24627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:31 dist-test-slave-s3n3 krb5kdc[24387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244751, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:31.358680 24627 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:12:31.360615 24627 webserver.cc:460] Webserver started at http://127.6.2.194:40303/ using document root <none> and password file <none>
I20260924 10:12:31.361387 24627 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:31.361538 24627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:31.361941 24627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:31.365007 24627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7"
format_stamp: "Formatted at 2026-09-24 10:12:31 on dist-test-slave-s3n3"
server_key: "4bdd3132b4baba83f936ee08d784871f"
server_key_iv: "2229a7ed1b059606341a73787fd8b84f"
server_key_version: "encryptionkey@0"
I20260924 10:12:31.366236 24627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7"
format_stamp: "Formatted at 2026-09-24 10:12:31 on dist-test-slave-s3n3"
server_key: "4bdd3132b4baba83f936ee08d784871f"
server_key_iv: "2229a7ed1b059606341a73787fd8b84f"
server_key_version: "encryptionkey@0"
I20260924 10:12:31.373005 24627 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260924 10:12:31.376804 24642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:31.378217 24627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:12:31.378399 24627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7"
format_stamp: "Formatted at 2026-09-24 10:12:31 on dist-test-slave-s3n3"
server_key: "4bdd3132b4baba83f936ee08d784871f"
server_key_iv: "2229a7ed1b059606341a73787fd8b84f"
server_key_version: "encryptionkey@0"
I20260924 10:12:31.378609 24627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:31.401469 24627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:31.407173 24627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:31.407510 24627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:31.408610 24627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:31.410132 24627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:31.410208 24627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:31.410326 24627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:31.410391 24627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:31.447043 24627 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:41431
I20260924 10:12:31.447146 24754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:41431 every 8 connection(s)
I20260924 10:12:31.448441 24627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:12:31.458284 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 24627
I20260924 10:12:31.458477 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:12:31.459010 6155 external_mini_cluster.cc:1428] Setting key 61f71b189e9090a9d31cc422fdaead35
Sep 24 10:12:31 dist-test-slave-s3n3 krb5kdc[24387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244751, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:12:31.470561 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:31.451134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:44467 (local address 127.6.2.254:44193)
0924 10:12:31.451605 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:31.451610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:31.452435 (+ 825us) server_negotiation.cc:407] Connection header received
0924 10:12:31.454320 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:31.454325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:31.454425 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:31.454548 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:31.456145 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:31.456877 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:31.458799 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:31.459333 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:31.459456 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:31.463776 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:31.463781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:31.463786 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:31.463821 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:31.467048 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:31.467923 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:31.467931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:31.467940 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:31.468080 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:31.468706 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:31.468710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:31.468712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:31.468990 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:31.469170 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:31.469883 (+ 713us) server_negotiation.cc:299] Negotiation successful
0924 10:12:31.470132 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":256,"threads_started":1}
I20260924 10:12:31.471635 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:31.451580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:44193 (local address 127.6.2.194:44467)
0924 10:12:31.452267 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:31.452307 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:31.454049 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:31.454782 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:31.454790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:31.455396 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:31.455967 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:31.455989 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:31.457043 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:31.457047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:31.458597 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:31.458614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:31.459582 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:31.459592 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:31.459855 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:31.460657 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:12:31.460710 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:31.463516 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:31.467281 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:31.467297 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:31.467313 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:31.467778 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:31.468262 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:31.468270 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:31.468277 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:31.468549 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:31.469127 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:31.469136 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:31.469516 (+ 380us) client_negotiation.cc:769] Sending connection context
0924 10:12:31.469784 (+ 268us) client_negotiation.cc:240] Negotiation successful
0924 10:12:31.470354 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":278,"threads_started":1}
I20260924 10:12:31.473397 24755 heartbeater.cc:347] Connected to a master server at 127.6.2.254:44193
I20260924 10:12:31.474138 24755 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:31.475438 24755 heartbeater.cc:511] Master 127.6.2.254:44193 requested a full tablet report, sending...
I20260924 10:12:31.477415 24432 ts_manager.cc:194] Registered new tserver with Master: 152a5ced4e2042d6a0f0fceeb10aa2b7 (127.6.2.194:41431)
I20260924 10:12:31.478338 24432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:44467
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:12:31.519302 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:44193
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:31.861653 24761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:31.861958 24761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:31.862123 24761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:31.869617 24761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:31.869745 24761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:31.869932 24761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:12:31.880201 24761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:44193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:31.881843 24761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:31.883725 24761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:31.897868 24766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:31.897766 24769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:31.897855 24767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:31.899296 24761 server_base.cc:1034] running on GCE node
I20260924 10:12:31.899987 24761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:31.901013 24761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:31.902259 24761 hybrid_clock.cc:648] HybridClock initialized: now 1790244751902196 us; error 57 us; skew 500 ppm
I20260924 10:12:31.902624 24761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:31 dist-test-slave-s3n3 krb5kdc[24387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244751, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:31.905972 24761 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:12:31.907606 24761 webserver.cc:460] Webserver started at http://127.6.2.195:43633/ using document root <none> and password file <none>
I20260924 10:12:31.908291 24761 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:31.908425 24761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:31.908738 24761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
W20260924 10:12:31.910281 24617 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.6.2.193:40303
I20260924 10:12:31.911717 24761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8"
format_stamp: "Formatted at 2026-09-24 10:12:31 on dist-test-slave-s3n3"
server_key: "8eaeb22967e9115458491860aacdc28b"
server_key_iv: "12af8f87d7bd845f015444a5ed71323c"
server_key_version: "encryptionkey@0"
I20260924 10:12:31.912524 24761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8"
format_stamp: "Formatted at 2026-09-24 10:12:31 on dist-test-slave-s3n3"
server_key: "8eaeb22967e9115458491860aacdc28b"
server_key_iv: "12af8f87d7bd845f015444a5ed71323c"
server_key_version: "encryptionkey@0"
I20260924 10:12:31.917927 24761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 10:12:31.921514 24776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:31.922909 24761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:12:31.923146 24761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8"
format_stamp: "Formatted at 2026-09-24 10:12:31 on dist-test-slave-s3n3"
server_key: "8eaeb22967e9115458491860aacdc28b"
server_key_iv: "12af8f87d7bd845f015444a5ed71323c"
server_key_version: "encryptionkey@0"
I20260924 10:12:31.923388 24761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:31.940191 24761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:31.941666 24621 heartbeater.cc:503] Master 127.6.2.254:44193 was elected leader, sending a full tablet report...
I20260924 10:12:31.945212 24761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:31.945528 24761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:31.946527 24761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:31.947927 24761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:31.948032 24761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:31.948180 24761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:31.948256 24761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:31.984123 24761 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:41211
I20260924 10:12:31.984169 24888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:41211 every 8 connection(s)
I20260924 10:12:31.985562 24761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:12:31.986052 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 24761
I20260924 10:12:31.986240 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:12:31.986841 6155 external_mini_cluster.cc:1428] Setting key a48498034dc33b7e7263324a80e7e8a1
Sep 24 10:12:31 dist-test-slave-s3n3 krb5kdc[24387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244751, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:12:32.009488 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:31.988924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:50611 (local address 127.6.2.254:44193)
0924 10:12:31.989425 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:31.989431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:31.991268 (+ 1837us) server_negotiation.cc:407] Connection header received
0924 10:12:31.993089 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:31.993094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:31.993164 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:31.993268 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:31.994959 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:31.996184 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:31.997954 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:31.998455 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:31.998738 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.003011 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:32.003020 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:32.003025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:32.003115 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:32.006263 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:32.007285 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:32.007290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:32.007293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:32.007433 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:32.007857 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:32.007860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:32.007862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:32.008144 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:32.008258 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:32.009018 (+ 760us) server_negotiation.cc:299] Negotiation successful
0924 10:12:32.009205 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":249,"threads_started":1}
I20260924 10:12:32.010466 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:31.989654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:44193 (local address 127.6.2.195:50611)
0924 10:12:31.991107 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:31.991164 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:31.992817 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:31.993488 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:31.993496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:31.994025 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:31.994603 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:31.994620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:31.996419 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:31.996424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:31.997760 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:31.997773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:31.998794 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:31.998799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:31.999003 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:31.999821 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:12:31.999849 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:32.002627 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:32.006519 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:32.006533 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:32.006554 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:32.007147 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:32.007552 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:32.007555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:32.007558 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:32.007745 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:32.008264 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:32.008271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:32.008637 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 10:12:32.008882 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 10:12:32.009301 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"thread_start_us":945,"threads_started":1}
I20260924 10:12:32.012768 24889 heartbeater.cc:347] Connected to a master server at 127.6.2.254:44193
I20260924 10:12:32.013144 24889 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:32.013962 24889 heartbeater.cc:511] Master 127.6.2.254:44193 requested a full tablet report, sending...
I20260924 10:12:32.015695 24432 ts_manager.cc:194] Registered new tserver with Master: f061ba4aa4d540b1bd3d5e4da58cc7c8 (127.6.2.195:41211)
I20260924 10:12:32.016647 24432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:50611
I20260924 10:12:32.029474 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:12:32.048477 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.033800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.6.2.254:44193)
0924 10:12:32.034013 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:32.034018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:32.034456 (+ 438us) server_negotiation.cc:407] Connection header received
0924 10:12:32.034644 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:32.034647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:32.034716 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:32.034806 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:32.036538 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.037316 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.038776 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.039286 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.039406 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.041936 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:32.041940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:32.041944 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:32.041977 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:32.045332 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:32.046281 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:32.046285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:32.046287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:32.046466 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:32.046924 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:32.046927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:32.046929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:32.047340 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:32.047569 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:32.047932 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 10:12:32.048106 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:12:32.052624 24432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:12:32.060803 24432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:12:32.086956 24902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.074827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:41211 (local address 127.0.0.1:44448)
0924 10:12:32.075816 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:32.075862 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:32.076055 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:32.077823 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:32.077832 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:32.077868 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:32.078433 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:32.078452 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.081432 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.081441 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:32.083988 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:32.084007 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.085933 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.085941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:32.086157 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.086214 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 10:12:32.086327 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 10:12:32.086552 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":2228,"threads_started":1}
I20260924 10:12:32.087829 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.074931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:41431 (local address 127.0.0.1:36806)
0924 10:12:32.078594 (+ 3663us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:32.078611 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:32.078827 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:32.081244 (+ 2417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:32.081249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:32.081275 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:32.081702 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:32.081716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.083585 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.083590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:32.085253 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:32.085265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.087007 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.087014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:32.087219 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.087264 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 10:12:32.087348 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:12:32.087481 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3511,"thread_start_us":3595,"threads_started":1}
I20260924 10:12:32.088140 24904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.075015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.6.2.195:41211)
0924 10:12:32.077153 (+ 2138us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:32.077160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:32.077193 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:12:32.077292 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:32.077299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:32.077540 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:32.078789 (+ 1249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:32.079132 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.081250 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.084171 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.085672 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.087311 (+ 1639us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.087549 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:32.087635 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 10:12:32.087815 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1989,"thread_start_us":1823,"threads_started":1}
I20260924 10:12:32.088584 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.075476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.6.2.194:41431)
0924 10:12:32.076659 (+ 1183us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:32.076671 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:32.078707 (+ 2036us) server_negotiation.cc:407] Connection header received
0924 10:12:32.079045 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:32.079052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:32.080813 (+ 1761us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:32.081003 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:32.081913 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.083426 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.085425 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.086785 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.086942 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.087190 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:32.088066 (+ 876us) server_negotiation.cc:299] Negotiation successful
0924 10:12:32.088243 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":924,"threads_started":1}
I20260924 10:12:32.088856 24903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.075413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:40303 (local address 127.0.0.1:58268)
0924 10:12:32.078807 (+ 3394us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:32.078826 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:32.078949 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:32.079714 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:32.079718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:32.079741 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:32.080149 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:32.080161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.083258 (+ 3097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.083264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:32.085776 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:32.085792 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.088090 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.088100 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:32.088276 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.088306 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:12:32.088417 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 10:12:32.088546 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3268,"thread_start_us":2832,"threads_started":1}
I20260924 10:12:32.089587 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.076181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.6.2.193:40303)
0924 10:12:32.078083 (+ 1902us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:32.078090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:32.079093 (+ 1003us) server_negotiation.cc:407] Connection header received
0924 10:12:32.079201 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:32.079209 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:32.079437 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:32.079622 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:32.080382 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.083129 (+ 2747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.085989 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.087586 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.087713 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.087959 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:32.089078 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0924 10:12:32.089243 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1735,"thread_start_us":1558,"threads_started":1}
I20260924 10:12:32.092286 24690 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7358aea82b4dea8692ed9afca71efb (DEFAULT_TABLE table=test-table [id=5b22f859242944b6867ff42ce17c8dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:32.092286 24824 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7358aea82b4dea8692ed9afca71efb (DEFAULT_TABLE table=test-table [id=5b22f859242944b6867ff42ce17c8dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:32.092947 24556 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7358aea82b4dea8692ed9afca71efb (DEFAULT_TABLE table=test-table [id=5b22f859242944b6867ff42ce17c8dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:32.093770 24824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7358aea82b4dea8692ed9afca71efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:32.093770 24690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7358aea82b4dea8692ed9afca71efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:32.094154 24556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7358aea82b4dea8692ed9afca71efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:32.106940 24908 tablet_bootstrap.cc:492] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283: Bootstrap starting.
I20260924 10:12:32.109880 24909 tablet_bootstrap.cc:492] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7: Bootstrap starting.
I20260924 10:12:32.111820 24910 tablet_bootstrap.cc:492] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8: Bootstrap starting.
I20260924 10:12:32.113582 24909 tablet_bootstrap.cc:654] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:32.113602 24908 tablet_bootstrap.cc:654] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:32.114845 24910 tablet_bootstrap.cc:654] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:32.114902 24909 log.cc:824] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:32.114908 24908 log.cc:824] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:32.116022 24910 log.cc:824] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:32.118310 24908 tablet_bootstrap.cc:492] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283: No bootstrap required, opened a new log
I20260924 10:12:32.118546 24909 tablet_bootstrap.cc:492] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7: No bootstrap required, opened a new log
I20260924 10:12:32.118661 24908 ts_tablet_manager.cc:1397] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260924 10:12:32.118942 24910 tablet_bootstrap.cc:492] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8: No bootstrap required, opened a new log
I20260924 10:12:32.119295 24910 ts_tablet_manager.cc:1397] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260924 10:12:32.119377 24909 ts_tablet_manager.cc:1397] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 10:12:32.126264 24910 raft_consensus.cc:348] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41211 } } peers { permanent_uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41431 } } peers { permanent_uuid: "f383722ea9f242428151cb9ec237f283" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40303 } }
I20260924 10:12:32.126694 24910 raft_consensus.cc:374] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:32.126808 24910 raft_consensus.cc:729] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f061ba4aa4d540b1bd3d5e4da58cc7c8, State: Initialized, Role: FOLLOWER
I20260924 10:12:32.127645 24910 consensus_queue.cc:260] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41211 } } peers { permanent_uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41431 } } peers { permanent_uuid: "f383722ea9f242428151cb9ec237f283" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40303 } }
I20260924 10:12:32.128925 24889 heartbeater.cc:503] Master 127.6.2.254:44193 was elected leader, sending a full tablet report...
I20260924 10:12:32.129536 24910 ts_tablet_manager.cc:1428] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20260924 10:12:32.129891 24908 raft_consensus.cc:348] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41211 } } peers { permanent_uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41431 } } peers { permanent_uuid: "f383722ea9f242428151cb9ec237f283" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40303 } }
I20260924 10:12:32.130409 24908 raft_consensus.cc:374] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:32.130527 24908 raft_consensus.cc:729] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f383722ea9f242428151cb9ec237f283, State: Initialized, Role: FOLLOWER
I20260924 10:12:32.131282 24908 consensus_queue.cc:260] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41211 } } peers { permanent_uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41431 } } peers { permanent_uuid: "f383722ea9f242428151cb9ec237f283" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40303 } }
I20260924 10:12:32.131366 24909 raft_consensus.cc:348] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41211 } } peers { permanent_uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41431 } } peers { permanent_uuid: "f383722ea9f242428151cb9ec237f283" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40303 } }
I20260924 10:12:32.131790 24909 raft_consensus.cc:374] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:32.131937 24909 raft_consensus.cc:729] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152a5ced4e2042d6a0f0fceeb10aa2b7, State: Initialized, Role: FOLLOWER
I20260924 10:12:32.132637 24909 consensus_queue.cc:260] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41211 } } peers { permanent_uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41431 } } peers { permanent_uuid: "f383722ea9f242428151cb9ec237f283" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40303 } }
I20260924 10:12:32.133179 24908 ts_tablet_manager.cc:1428] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 10:12:32.139772 24755 heartbeater.cc:503] Master 127.6.2.254:44193 was elected leader, sending a full tablet report...
I20260924 10:12:32.140587 24909 ts_tablet_manager.cc:1428] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7: Time spent starting tablet: real 0.021s user 0.010s sys 0.008s
W20260924 10:12:32.160379 24622 tablet.cc:2369] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:12:32.200621 24756 tablet.cc:2369] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:12:32.201066 24914 raft_consensus.cc:482] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:12:32.201426 24914 raft_consensus.cc:504] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41211 } } peers { permanent_uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41431 } } peers { permanent_uuid: "f383722ea9f242428151cb9ec237f283" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40303 } }
I20260924 10:12:32.203285 24914 leader_election.cc:288] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 152a5ced4e2042d6a0f0fceeb10aa2b7 (127.6.2.194:41431), f383722ea9f242428151cb9ec237f283 (127.6.2.193:40303)
I20260924 10:12:32.210131 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.203802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:36651 (local address 127.6.2.194:41431)
0924 10:12:32.204007 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:32.204012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:32.204112 (+ 100us) server_negotiation.cc:407] Connection header received
0924 10:12:32.204326 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:32.204330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:32.204406 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:32.204516 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:32.205047 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.206030 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.207992 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.209237 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.209385 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.209474 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:32.209812 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 10:12:32.209930 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:12:32.210181 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.203783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:41431 (local address 127.6.2.195:36651)
0924 10:12:32.203997 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:32.204019 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:32.204155 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:32.204579 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:32.204587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:32.204631 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:32.204905 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:32.204927 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.206157 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.206162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:32.207844 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:32.207858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.209412 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.209417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:32.209608 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.209644 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:12:32.209757 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 10:12:32.209907 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 10:12:32.211493 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.203783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:40303 (local address 127.6.2.195:45437)
0924 10:12:32.205049 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:32.205066 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:32.205191 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:32.205550 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:32.205555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:32.205576 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:32.205872 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:32.205880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.207329 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.207333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:32.209236 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:32.209250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.210844 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.210848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:32.210962 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.210984 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:12:32.211134 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 10:12:32.211232 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"mutex_wait_us":474,"thread_start_us":423,"threads_started":1}
I20260924 10:12:32.211552 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.203832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:45437 (local address 127.6.2.193:40303)
0924 10:12:32.204036 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:32.204043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:32.205169 (+ 1126us) server_negotiation.cc:407] Connection header received
0924 10:12:32.205336 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:32.205340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:32.205421 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:32.205540 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:32.206025 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.207150 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.209411 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.210645 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.210774 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.210875 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:32.211158 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 10:12:32.211293 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:12:32.211357 24710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7358aea82b4dea8692ed9afca71efb" candidate_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7" is_pre_election: true
I20260924 10:12:32.211970 24710 raft_consensus.cc:2454] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f061ba4aa4d540b1bd3d5e4da58cc7c8 in term 0.
I20260924 10:12:32.212707 24780 leader_election.cc:302] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152a5ced4e2042d6a0f0fceeb10aa2b7, f061ba4aa4d540b1bd3d5e4da58cc7c8; no voters:
I20260924 10:12:32.212664 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7358aea82b4dea8692ed9afca71efb" candidate_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f383722ea9f242428151cb9ec237f283" is_pre_election: true
I20260924 10:12:32.213145 24914 raft_consensus.cc:2781] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:12:32.213238 24576 raft_consensus.cc:2454] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f061ba4aa4d540b1bd3d5e4da58cc7c8 in term 0.
I20260924 10:12:32.213320 24914 raft_consensus.cc:482] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:12:32.213424 24914 raft_consensus.cc:3037] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:32.215781 24914 raft_consensus.cc:504] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41211 } } peers { permanent_uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41431 } } peers { permanent_uuid: "f383722ea9f242428151cb9ec237f283" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40303 } }
I20260924 10:12:32.216359 24914 leader_election.cc:288] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [CANDIDATE]: Term 1 election: Requested vote from peers 152a5ced4e2042d6a0f0fceeb10aa2b7 (127.6.2.194:41431), f383722ea9f242428151cb9ec237f283 (127.6.2.193:40303)
I20260924 10:12:32.216825 24710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7358aea82b4dea8692ed9afca71efb" candidate_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7"
I20260924 10:12:32.216822 24576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7358aea82b4dea8692ed9afca71efb" candidate_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f383722ea9f242428151cb9ec237f283"
I20260924 10:12:32.217043 24710 raft_consensus.cc:3037] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:32.217047 24576 raft_consensus.cc:3037] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:32.220306 24710 raft_consensus.cc:2454] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f061ba4aa4d540b1bd3d5e4da58cc7c8 in term 1.
I20260924 10:12:32.220297 24576 raft_consensus.cc:2454] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f061ba4aa4d540b1bd3d5e4da58cc7c8 in term 1.
I20260924 10:12:32.220731 24780 leader_election.cc:302] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152a5ced4e2042d6a0f0fceeb10aa2b7, f061ba4aa4d540b1bd3d5e4da58cc7c8; no voters:
I20260924 10:12:32.221038 24914 raft_consensus.cc:2781] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:32.221521 24914 raft_consensus.cc:686] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [term 1 LEADER]: Becoming Leader. State: Replica: f061ba4aa4d540b1bd3d5e4da58cc7c8, State: Running, Role: LEADER
I20260924 10:12:32.222090 24914 consensus_queue.cc:237] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41211 } } peers { permanent_uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41431 } } peers { permanent_uuid: "f383722ea9f242428151cb9ec237f283" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40303 } }
I20260924 10:12:32.227154 24431 catalog_manager.cc:5495] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f061ba4aa4d540b1bd3d5e4da58cc7c8 (127.6.2.195). New cstate: current_term: 1 leader_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f061ba4aa4d540b1bd3d5e4da58cc7c8" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f383722ea9f242428151cb9ec237f283" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40303 } health_report { overall_health: UNKNOWN } } }
W20260924 10:12:32.237504 24890 tablet.cc:2369] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:12:32.249300 24904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.243780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44458 (local address 127.6.2.195:41211)
0924 10:12:32.244050 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:32.244056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:32.244075 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:12:32.244282 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:32.244286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:32.244363 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:32.244517 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:12:32.244927 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.245626 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.247028 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.247490 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.247621 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.248215 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:12:32.248710 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:12:32.248904 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:32.249018 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 10:12:32.249066 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 10:12:32.257853 24710 raft_consensus.cc:1261] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7 [term 1 FOLLOWER]: Refusing update from remote peer f061ba4aa4d540b1bd3d5e4da58cc7c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:12:32.257851 24576 raft_consensus.cc:1261] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283 [term 1 FOLLOWER]: Refusing update from remote peer f061ba4aa4d540b1bd3d5e4da58cc7c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:12:32.258842 24918 consensus_queue.cc:1035] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "152a5ced4e2042d6a0f0fceeb10aa2b7" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:12:32.259231 24914 consensus_queue.cc:1035] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f383722ea9f242428151cb9ec237f283" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 40303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:12:32.288566 24432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:12:32.288858 24432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:12:32.292346 24432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b233dcb2985b42f2b41493f9fd06036e: Sending DeleteTablet for 3 replicas of tablet 6a7358aea82b4dea8692ed9afca71efb
I20260924 10:12:32.293497 24824 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7358aea82b4dea8692ed9afca71efb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:32 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:44448
I20260924 10:12:32.293552 24556 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7358aea82b4dea8692ed9afca71efb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:32 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:58268
I20260924 10:12:32.293552 24690 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7358aea82b4dea8692ed9afca71efb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:32 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:36806
I20260924 10:12:32.299712 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 24493
I20260924 10:12:32.314005 24931 tablet_replica.cc:330] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283: stopping tablet replica
I20260924 10:12:32.314736 24931 raft_consensus.cc:2229] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:12:32.315510 24931 raft_consensus.cc:2258] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:32.316365 24933 tablet_replica.cc:330] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8: stopping tablet replica
I20260924 10:12:32.316927 24933 raft_consensus.cc:2229] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:12:32.318780 24931 ts_tablet_manager.cc:1905] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:32.318792 24933 raft_consensus.cc:2258] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:32.319042 24932 tablet_replica.cc:330] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7: stopping tablet replica
I20260924 10:12:32.319734 24932 raft_consensus.cc:2229] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7 [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 24 10:12:32 dist-test-slave-s3n3 krb5kdc[24387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:12:32.320859 24933 ts_tablet_manager.cc:1905] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:32.321203 24932 raft_consensus.cc:2258] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:32.328188 24931 ts_tablet_manager.cc:1918] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:32.328646 24933 ts_tablet_manager.cc:1918] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:32.328809 24933 log.cc:1196] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/6a7358aea82b4dea8692ed9afca71efb
I20260924 10:12:32.329401 24933 ts_tablet_manager.cc:1939] T 6a7358aea82b4dea8692ed9afca71efb P f061ba4aa4d540b1bd3d5e4da58cc7c8: Deleting consensus metadata
I20260924 10:12:32.330827 24932 ts_tablet_manager.cc:1905] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:32.331420 24417 catalog_manager.cc:4841] TS f061ba4aa4d540b1bd3d5e4da58cc7c8 (127.6.2.195:41211): tablet 6a7358aea82b4dea8692ed9afca71efb (table test-table [id=5b22f859242944b6867ff42ce17c8dab]) successfully deleted
I20260924 10:12:32.329895 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.301732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.6.2.193:40303)
0924 10:12:32.301922 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:32.301929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:32.303690 (+ 1761us) server_negotiation.cc:407] Connection header received
0924 10:12:32.304745 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:32.304754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:32.304861 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:32.304964 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:32.309270 (+ 4306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.310920 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.313608 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.314597 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.314740 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.321932 (+ 7192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:32.321946 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:32.321966 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:32.322013 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:32.326095 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:32.327039 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:32.327048 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:32.327052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:32.327296 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:32.327744 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:32.327749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:32.327752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:32.328225 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:32.328529 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:32.329219 (+ 690us) server_negotiation.cc:299] Negotiation successful
0924 10:12:32.329481 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:12:32.331871 24931 log.cc:1196] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/6a7358aea82b4dea8692ed9afca71efb
I20260924 10:12:32.332629 24931 ts_tablet_manager.cc:1939] T 6a7358aea82b4dea8692ed9afca71efb P f383722ea9f242428151cb9ec237f283: Deleting consensus metadata
I20260924 10:12:32.332753 24616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:32.338941 24932 ts_tablet_manager.cc:1918] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:32.339126 24932 log.cc:1196] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/6a7358aea82b4dea8692ed9afca71efb
I20260924 10:12:32.339653 24932 ts_tablet_manager.cc:1939] T 6a7358aea82b4dea8692ed9afca71efb P 152a5ced4e2042d6a0f0fceeb10aa2b7: Deleting consensus metadata
I20260924 10:12:32.341496 24419 catalog_manager.cc:4841] TS 152a5ced4e2042d6a0f0fceeb10aa2b7 (127.6.2.194:41431): tablet 6a7358aea82b4dea8692ed9afca71efb (table test-table [id=5b22f859242944b6867ff42ce17c8dab]) successfully deleted
I20260924 10:12:32.428228 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 24493
I20260924 10:12:32.429557 24416 catalog_manager.cc:4841] TS f383722ea9f242428151cb9ec237f283 (127.6.2.193:40303): tablet 6a7358aea82b4dea8692ed9afca71efb (table test-table [id=5b22f859242944b6867ff42ce17c8dab]) successfully deleted
I20260924 10:12:32.452749 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 24627
Sep 24 10:12:32 dist-test-slave-s3n3 krb5kdc[24387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:12:32.473466 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.453586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.6.2.194:41431)
0924 10:12:32.454249 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:32.454256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:32.454276 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:12:32.454338 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:32.454344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:32.454462 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:32.454591 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:32.456289 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.457571 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.459656 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.460357 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.460488 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.466185 (+ 5697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:32.466200 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:32.466220 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:32.466263 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:32.470422 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:32.471300 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:32.471310 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:32.471314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:32.471468 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:32.471821 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:32.471826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:32.471829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:32.472317 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:32.472611 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:32.472820 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 10:12:32.473060 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519}
I20260924 10:12:32.474146 24750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:32.565799 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 24627
I20260924 10:12:32.583611 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 24761
Sep 24 10:12:32 dist-test-slave-s3n3 krb5kdc[24387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:12:32.600502 24904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:32.584456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.6.2.195:41211)
0924 10:12:32.584633 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:32.584639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:32.584688 (+ 49us) server_negotiation.cc:407] Connection header received
0924 10:12:32.584923 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:32.584927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:32.584994 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:32.585107 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:32.586497 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.587858 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.589206 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:32.589669 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:32.589814 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:32.594145 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:32.594157 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:32.594173 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:32.594211 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:32.597533 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:32.598266 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:32.598273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:32.598275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:32.598428 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:32.598839 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:32.598844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:32.598846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:32.599273 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:32.599616 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:32.599881 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 10:12:32.600104 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:12:32.601210 24884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:32.727850 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 24761
I20260924 10:12:32.748616 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 24400
I20260924 10:12:32.749730 24462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:32.863559 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 24400
2026-09-24T10:12:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4437 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T10:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:32Z Disabled control of system clock
I20260924 10:12:32.941996 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:33749
--webserver_interface=127.6.2.254
--webserver_port=0
--builtin_ntp_servers=127.6.2.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:33749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:12:33.280166 24942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:33.280455 24942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:33.280622 24942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:33.288093 24942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:33.288209 24942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:33.288281 24942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:33.288369 24942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:33.288430 24942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:33.298147 24942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:33749
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:33749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:33.299553 24942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:33.301429 24942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:33.313373 24947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:33.313441 24950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:33.313417 24948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:33.313997 24942 server_base.cc:1034] running on GCE node
I20260924 10:12:33.314864 24942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:33.316454 24942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:33.317693 24942 hybrid_clock.cc:648] HybridClock initialized: now 1790244753317636 us; error 40 us; skew 500 ppm
I20260924 10:12:33.318049 24942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:12:33.320480 24942 webserver.cc:460] Webserver started at http://127.6.2.254:44489/ using document root <none> and password file <none>
I20260924 10:12:33.321259 24942 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:33.321401 24942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:33.321741 24942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:33.324915 24942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "fcde56466a9d472c8c09e0dea1b746c9"
format_stamp: "Formatted at 2026-09-24 10:12:33 on dist-test-slave-s3n3"
server_key: "57747a3b8de9dd6705e7da6fd0fe527e"
server_key_iv: "9b559f4c52d068676dd46039cf9c4408"
server_key_version: "encryptionkey@0"
I20260924 10:12:33.325731 24942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "fcde56466a9d472c8c09e0dea1b746c9"
format_stamp: "Formatted at 2026-09-24 10:12:33 on dist-test-slave-s3n3"
server_key: "57747a3b8de9dd6705e7da6fd0fe527e"
server_key_iv: "9b559f4c52d068676dd46039cf9c4408"
server_key_version: "encryptionkey@0"
I20260924 10:12:33.331111 24942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 10:12:33.334928 24956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:33.336392 24942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:12:33.336557 24942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "fcde56466a9d472c8c09e0dea1b746c9"
format_stamp: "Formatted at 2026-09-24 10:12:33 on dist-test-slave-s3n3"
server_key: "57747a3b8de9dd6705e7da6fd0fe527e"
server_key_iv: "9b559f4c52d068676dd46039cf9c4408"
server_key_version: "encryptionkey@0"
I20260924 10:12:33.336745 24942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:33.359266 24942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:33.360203 24942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:33.360530 24942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:33.378486 24942 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:33749
I20260924 10:12:33.378507 25007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:33749 every 8 connection(s)
I20260924 10:12:33.379864 24942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:33.383742 25008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:33.387404 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 24942
I20260924 10:12:33.387596 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:33.388077 6155 external_mini_cluster.cc:1428] Setting key 7d5e5011a7c3f74d2fcdf045fad47854
I20260924 10:12:33.394927 25008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9: Bootstrap starting.
I20260924 10:12:33.398658 25008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:33.400007 25008 log.cc:824] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:33.402930 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:33.390207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.6.2.254:33749)
0924 10:12:33.391268 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:33.391279 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:33.391327 (+ 48us) server_negotiation.cc:407] Connection header received
0924 10:12:33.393437 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:33.393460 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:33.393931 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:33.394252 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:33.394898 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:33.396315 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:33.398291 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:33.399174 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:33.399325 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:33.400640 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:33.400648 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:33.400661 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:33.401235 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:33.401348 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:33.401369 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:33.401488 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:33.401730 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 10:12:33.402046 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":361,"threads_started":1}
I20260924 10:12:33.403067 25008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9: No bootstrap required, opened a new log
I20260924 10:12:33.411034 25008 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde56466a9d472c8c09e0dea1b746c9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33749 } }
I20260924 10:12:33.411481 25008 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:33.411576 25008 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcde56466a9d472c8c09e0dea1b746c9, State: Initialized, Role: FOLLOWER
I20260924 10:12:33.412171 25008 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde56466a9d472c8c09e0dea1b746c9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33749 } }
I20260924 10:12:33.412425 25008 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:33.412560 25008 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:33.412737 25008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:33.415448 25008 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde56466a9d472c8c09e0dea1b746c9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33749 } }
I20260924 10:12:33.415973 25008 leader_election.cc:302] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcde56466a9d472c8c09e0dea1b746c9; no voters:
I20260924 10:12:33.416605 25008 leader_election.cc:288] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:33.416771 25013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:33.417486 25013 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [term 1 LEADER]: Becoming Leader. State: Replica: fcde56466a9d472c8c09e0dea1b746c9, State: Running, Role: LEADER
I20260924 10:12:33.418155 25013 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde56466a9d472c8c09e0dea1b746c9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33749 } }
I20260924 10:12:33.418872 25008 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:33.420866 25014 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcde56466a9d472c8c09e0dea1b746c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde56466a9d472c8c09e0dea1b746c9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33749 } } }
I20260924 10:12:33.421401 25014 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:33.420967 25015 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcde56466a9d472c8c09e0dea1b746c9. Latest consensus state: current_term: 1 leader_uuid: "fcde56466a9d472c8c09e0dea1b746c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde56466a9d472c8c09e0dea1b746c9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 33749 } } }
I20260924 10:12:33.421515 25015 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:33.423708 25020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:33.429253 25020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:33.438107 25020 catalog_manager.cc:1347] Generated new cluster ID: d7d2f35754b34b17834ba92711e687e9
I20260924 10:12:33.438287 25020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:33.472630 25020 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:33.474027 25020 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:33.489028 25020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9: Generated new TSK 0
I20260924 10:12:33.489980 25020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:12:33.501924 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33749
--builtin_ntp_servers=127.6.2.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:12:33.837981 25032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:33.838277 25032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:33.838440 25032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:33.846125 25032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:33.846243 25032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:33.846424 25032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:12:33.855767 25032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:33.857147 25032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:33.858990 25032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:33.872975 25037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:33.872896 25038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:33.872881 25040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:33.874315 25032 server_base.cc:1034] running on GCE node
I20260924 10:12:33.875018 25032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:33.876107 25032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:33.877362 25032 hybrid_clock.cc:648] HybridClock initialized: now 1790244753877295 us; error 67 us; skew 500 ppm
I20260924 10:12:33.877744 25032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:12:33.885295 25032 webserver.cc:460] Webserver started at http://127.6.2.193:43641/ using document root <none> and password file <none>
I20260924 10:12:33.886365 25032 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:33.886534 25032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:33.886945 25032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:33.889973 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "fd71633a50f94411af879d456bdb6470"
format_stamp: "Formatted at 2026-09-24 10:12:33 on dist-test-slave-s3n3"
server_key: "6bd31d7d5df838ef22515c32278fa1d2"
server_key_iv: "89b098563c790375d74d0ad46164c8e8"
server_key_version: "encryptionkey@0"
I20260924 10:12:33.890799 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "fd71633a50f94411af879d456bdb6470"
format_stamp: "Formatted at 2026-09-24 10:12:33 on dist-test-slave-s3n3"
server_key: "6bd31d7d5df838ef22515c32278fa1d2"
server_key_iv: "89b098563c790375d74d0ad46164c8e8"
server_key_version: "encryptionkey@0"
I20260924 10:12:33.896275 25032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:12:33.899919 25046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:33.901291 25032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:12:33.901527 25032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "fd71633a50f94411af879d456bdb6470"
format_stamp: "Formatted at 2026-09-24 10:12:33 on dist-test-slave-s3n3"
server_key: "6bd31d7d5df838ef22515c32278fa1d2"
server_key_iv: "89b098563c790375d74d0ad46164c8e8"
server_key_version: "encryptionkey@0"
I20260924 10:12:33.901777 25032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:33.926702 25032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:33.927659 25032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:33.927970 25032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:33.928952 25032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:33.930675 25032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:33.930809 25032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:33.930951 25032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:33.931042 25032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:33.965488 25032 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:42497
I20260924 10:12:33.965538 25158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:42497 every 8 connection(s)
I20260924 10:12:33.966786 25032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:12:33.970624 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 25032
I20260924 10:12:33.970813 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:12:33.971427 6155 external_mini_cluster.cc:1428] Setting key 41f9375777d212c5087b76180da58bf8
I20260924 10:12:33.974500 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33749
--builtin_ntp_servers=127.6.2.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:12:33.981297 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:33.969411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:55689 (local address 127.6.2.254:33749)
0924 10:12:33.969873 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:33.969878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:33.971114 (+ 1236us) server_negotiation.cc:407] Connection header received
0924 10:12:33.973335 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:33.973341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:33.973422 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:33.973533 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:33.974991 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:33.975805 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:33.978251 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:33.978650 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:33.978762 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:33.980462 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:33.980468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:33.980471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:33.980491 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:33.980534 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:33.980546 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:33.980620 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:33.980924 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 10:12:33.981011 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":247,"threads_started":1}
I20260924 10:12:33.982674 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:33.969780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:33749 (local address 127.6.2.193:55689)
0924 10:12:33.970890 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:33.970937 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:33.973014 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:33.973842 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:33.973853 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:33.973977 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:33.974812 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:33.974840 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:33.975961 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:33.975970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:33.978086 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:33.978104 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:33.978896 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:33.978905 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:33.979256 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:33.980081 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:12:33.980087 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:33.980143 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:12:33.980260 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:33.980650 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:33.980663 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:33.980673 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:12:33.980877 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 10:12:33.981306 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":282,"threads_started":1}
I20260924 10:12:33.984423 25159 heartbeater.cc:347] Connected to a master server at 127.6.2.254:33749
I20260924 10:12:33.984772 25159 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:33.985522 25159 heartbeater.cc:511] Master 127.6.2.254:33749 requested a full tablet report, sending...
I20260924 10:12:33.987845 24973 ts_manager.cc:194] Registered new tserver with Master: fd71633a50f94411af879d456bdb6470 (127.6.2.193:42497)
I20260924 10:12:33.990132 24973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.2.193:55689
W20260924 10:12:34.317256 25163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:34.317550 25163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:34.317638 25163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:34.325363 25163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:34.325470 25163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:34.325577 25163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:12:34.334808 25163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:34.336488 25163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:34.338353 25163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:34.352054 25168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:34.352114 25171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:34.352155 25169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:34.352998 25163 server_base.cc:1034] running on GCE node
I20260924 10:12:34.353624 25163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:34.354640 25163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:34.355894 25163 hybrid_clock.cc:648] HybridClock initialized: now 1790244754355835 us; error 87 us; skew 500 ppm
I20260924 10:12:34.356259 25163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:12:34.358574 25163 webserver.cc:460] Webserver started at http://127.6.2.194:45007/ using document root <none> and password file <none>
I20260924 10:12:34.359391 25163 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:34.359521 25163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:34.359838 25163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:34.362680 25163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "4471918e8a3f479a980122b773ae8e56"
format_stamp: "Formatted at 2026-09-24 10:12:34 on dist-test-slave-s3n3"
server_key: "fa78b554009ae5709a6acf1ad8d1edcb"
server_key_iv: "45f50f3dce5c376f4123b66eca6ba9f2"
server_key_version: "encryptionkey@0"
I20260924 10:12:34.363504 25163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "4471918e8a3f479a980122b773ae8e56"
format_stamp: "Formatted at 2026-09-24 10:12:34 on dist-test-slave-s3n3"
server_key: "fa78b554009ae5709a6acf1ad8d1edcb"
server_key_iv: "45f50f3dce5c376f4123b66eca6ba9f2"
server_key_version: "encryptionkey@0"
I20260924 10:12:34.368563 25163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:12:34.372006 25177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:34.373397 25163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:12:34.373605 25163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "4471918e8a3f479a980122b773ae8e56"
format_stamp: "Formatted at 2026-09-24 10:12:34 on dist-test-slave-s3n3"
server_key: "fa78b554009ae5709a6acf1ad8d1edcb"
server_key_iv: "45f50f3dce5c376f4123b66eca6ba9f2"
server_key_version: "encryptionkey@0"
I20260924 10:12:34.373837 25163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:34.388166 25163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:34.389027 25163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:34.389309 25163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:34.390288 25163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:34.391844 25163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:34.391964 25163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:34.392104 25163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:34.392189 25163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:34.425613 25163 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:41201
I20260924 10:12:34.425662 25289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:41201 every 8 connection(s)
I20260924 10:12:34.426985 25163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:12:34.434052 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 25163
I20260924 10:12:34.434226 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:12:34.434677 6155 external_mini_cluster.cc:1428] Setting key d0529f7e2ab0cf5ab040e530f2fbc7e1
I20260924 10:12:34.437477 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33749
--builtin_ntp_servers=127.6.2.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:12:34.440238 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:34.429631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:37029 (local address 127.6.2.254:33749)
0924 10:12:34.429894 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:34.429902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:34.431629 (+ 1727us) server_negotiation.cc:407] Connection header received
0924 10:12:34.433420 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:34.433426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:34.433517 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:34.433630 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:34.434839 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.435751 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.437303 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.437780 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.437915 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:34.439380 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:34.439386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:34.439389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:34.439408 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:34.439455 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:34.439466 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:34.439559 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:34.439821 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 10:12:34.439920 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:12:34.441529 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:34.430352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:33749 (local address 127.6.2.194:37029)
0924 10:12:34.431462 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:34.431504 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:34.433155 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:34.433872 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:34.433880 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:34.433964 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:34.434629 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:34.434648 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.435901 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.435907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:34.437150 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:34.437162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.438020 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.438030 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:34.438256 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:34.439004 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:12:34.439007 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:34.439048 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:12:34.439185 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:34.439575 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:34.439587 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:34.439597 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:12:34.439801 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 10:12:34.440204 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":424,"threads_started":1}
I20260924 10:12:34.442955 25290 heartbeater.cc:347] Connected to a master server at 127.6.2.254:33749
I20260924 10:12:34.443293 25290 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:34.444073 25290 heartbeater.cc:511] Master 127.6.2.254:33749 requested a full tablet report, sending...
I20260924 10:12:34.445789 24973 ts_manager.cc:194] Registered new tserver with Master: 4471918e8a3f479a980122b773ae8e56 (127.6.2.194:41201)
I20260924 10:12:34.447049 24973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.2.194:37029
W20260924 10:12:34.779111 25293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:34.779428 25293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:34.779516 25293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:34.788254 25293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:34.788384 25293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:34.788504 25293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:12:34.798854 25293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:33749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:34.800287 25293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:34.802135 25293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:34.816205 25298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:34.816375 25301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:34.816375 25299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:34.817236 25293 server_base.cc:1034] running on GCE node
I20260924 10:12:34.817936 25293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:34.819031 25293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:34.820309 25293 hybrid_clock.cc:648] HybridClock initialized: now 1790244754820240 us; error 63 us; skew 500 ppm
I20260924 10:12:34.820649 25293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:12:34.823177 25293 webserver.cc:460] Webserver started at http://127.6.2.195:40747/ using document root <none> and password file <none>
I20260924 10:12:34.823944 25293 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:34.824070 25293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:34.824414 25293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:34.827653 25293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "8e78276b476843e2bc44be8a74b3d6e4"
format_stamp: "Formatted at 2026-09-24 10:12:34 on dist-test-slave-s3n3"
server_key: "550a92adf5a1f76c6ee90d25c9a0e5e8"
server_key_iv: "777ad829a300f61a861090fa4a2e11b3"
server_key_version: "encryptionkey@0"
I20260924 10:12:34.828473 25293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "8e78276b476843e2bc44be8a74b3d6e4"
format_stamp: "Formatted at 2026-09-24 10:12:34 on dist-test-slave-s3n3"
server_key: "550a92adf5a1f76c6ee90d25c9a0e5e8"
server_key_iv: "777ad829a300f61a861090fa4a2e11b3"
server_key_version: "encryptionkey@0"
I20260924 10:12:34.833884 25293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:12:34.837531 25307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:34.839052 25293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 10:12:34.839327 25293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "8e78276b476843e2bc44be8a74b3d6e4"
format_stamp: "Formatted at 2026-09-24 10:12:34 on dist-test-slave-s3n3"
server_key: "550a92adf5a1f76c6ee90d25c9a0e5e8"
server_key_iv: "777ad829a300f61a861090fa4a2e11b3"
server_key_version: "encryptionkey@0"
I20260924 10:12:34.839577 25293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:34.856276 25293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:34.857143 25293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:34.857436 25293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:34.858457 25293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:34.860052 25293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:34.860172 25293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:34.860307 25293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:34.860405 25293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:34.894954 25293 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:42433
I20260924 10:12:34.895044 25419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:42433 every 8 connection(s)
I20260924 10:12:34.896327 25293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:12:34.904686 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 25293
I20260924 10:12:34.904881 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:12:34.905385 6155 external_mini_cluster.cc:1428] Setting key 7f20b887df8bdd4644c3270fe38acfc2
I20260924 10:12:34.909646 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:34.899156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:38635 (local address 127.6.2.254:33749)
0924 10:12:34.899369 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:34.899375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:34.900727 (+ 1352us) server_negotiation.cc:407] Connection header received
0924 10:12:34.902495 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:34.902500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:34.902593 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:34.902693 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:34.903876 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.904794 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.906725 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.907370 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.907516 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:34.908948 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:34.908953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:34.908957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:34.908979 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:34.909013 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:34.909024 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:34.909096 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:34.909313 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 10:12:34.909391 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:12:34.910705 25422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:34.899683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:33749 (local address 127.6.2.195:38635)
0924 10:12:34.900566 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:34.900607 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:34.902218 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:34.902909 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:34.902917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:34.903018 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:34.903663 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:34.903681 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.904968 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.904974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:34.906560 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:34.906579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.907586 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.907591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:34.907794 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:34.908620 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:12:34.908624 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:34.908664 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:12:34.908745 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:34.909119 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:34.909127 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:34.909134 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:12:34.909286 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 10:12:34.909600 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":299,"threads_started":1}
I20260924 10:12:34.912055 25420 heartbeater.cc:347] Connected to a master server at 127.6.2.254:33749
I20260924 10:12:34.912312 25420 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:34.913012 25420 heartbeater.cc:511] Master 127.6.2.254:33749 requested a full tablet report, sending...
I20260924 10:12:34.914443 24973 ts_manager.cc:194] Registered new tserver with Master: 8e78276b476843e2bc44be8a74b3d6e4 (127.6.2.195:42433)
I20260924 10:12:34.915356 24973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.2.195:38635
I20260924 10:12:34.920953 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:12:34.932482 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:34.925569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.6.2.254:33749)
0924 10:12:34.925738 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:34.925742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:34.926186 (+ 444us) server_negotiation.cc:407] Connection header received
0924 10:12:34.926319 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:34.926323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:34.926409 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:34.926496 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:34.926985 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.927752 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.929410 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.930061 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.930224 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:34.931734 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:34.931739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:34.931743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:34.931762 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:34.931798 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:34.931810 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:34.931990 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:34.932208 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 10:12:34.932254 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:12:34.936789 24973 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:12:34.944068 24973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:12:34.969923 25433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:34.957351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:41201 (local address 127.0.0.1:38862)
0924 10:12:34.959706 (+ 2355us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:34.959729 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:34.959846 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:34.960815 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:34.960823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:34.960854 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:34.961293 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:34.961306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.963297 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.963302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:34.965816 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:34.965829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.968058 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.968065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:34.969119 (+ 1054us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:34.969183 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 10:12:34.969318 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 10:12:34.969510 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2235,"mutex_wait_us":1,"thread_start_us":785,"threads_started":1}
I20260924 10:12:34.970700 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:34.957930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.6.2.194:41201)
0924 10:12:34.959429 (+ 1499us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:34.959441 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:34.959979 (+ 538us) server_negotiation.cc:407] Connection header received
0924 10:12:34.960077 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:34.960084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:34.960355 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:34.960551 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:34.961524 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.963134 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.966051 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.967513 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.967649 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:34.967927 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:34.970184 (+ 2257us) server_negotiation.cc:299] Negotiation successful
0924 10:12:34.970373 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1312,"thread_start_us":1199,"threads_started":1}
I20260924 10:12:34.970708 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:34.956818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42497 (local address 127.0.0.1:41114)
0924 10:12:34.958565 (+ 1747us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:34.958613 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:34.958776 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:34.961703 (+ 2927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:34.961707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:34.961731 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:34.962120 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:34.962134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.965344 (+ 3210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.965349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:34.967444 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:34.967461 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.969990 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.969999 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:34.970184 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:34.970217 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:12:34.970281 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 10:12:34.970418 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1471,"thread_start_us":2619,"threads_started":1}
I20260924 10:12:34.972285 25432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:34.956838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.6.2.193:42497)
0924 10:12:34.959862 (+ 3024us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:34.959869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:34.959901 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:12:34.961211 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:34.961222 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:34.961470 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:34.962303 (+ 833us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:34.962533 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.965182 (+ 2649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.967707 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.969753 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.970841 (+ 1088us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:34.971680 (+ 839us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:34.971751 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 10:12:34.971908 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2861,"thread_start_us":2744,"threads_started":1}
I20260924 10:12:34.975405 25225 tablet_service.cc:1467] Processing CreateTablet for tablet 701f311c4d874d2e8c3804d32ae1340f (DEFAULT_TABLE table=test-table [id=f2245ec6ce2f46d1b922825a03bfcb17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:34.976750 25225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 701f311c4d874d2e8c3804d32ae1340f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:34.978057 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 701f311c4d874d2e8c3804d32ae1340f (DEFAULT_TABLE table=test-table [id=f2245ec6ce2f46d1b922825a03bfcb17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:34.979480 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 701f311c4d874d2e8c3804d32ae1340f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:34.983318 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:34.959894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:42433 (local address 127.0.0.1:59402)
0924 10:12:34.971153 (+ 11259us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:34.971176 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:34.971277 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:34.975448 (+ 4171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:34.975453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:34.975480 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:34.975655 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:34.975667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.978819 (+ 3152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.978824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:34.980688 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:34.980706 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.982490 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.982496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:34.982678 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:34.982716 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 10:12:34.982822 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 10:12:34.982976 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11138,"thread_start_us":14790,"threads_started":1}
I20260924 10:12:34.983652 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:34.960233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.6.2.195:42433)
0924 10:12:34.974822 (+ 14589us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:34.974830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:34.974884 (+ 54us) server_negotiation.cc:407] Connection header received
0924 10:12:34.974991 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:34.975005 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:34.975259 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:34.975459 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:34.977032 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.978586 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.980874 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:34.982307 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:34.982454 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:34.982716 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:34.982874 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 10:12:34.983046 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14387,"thread_start_us":14076,"threads_started":1}
I20260924 10:12:34.987598 25355 tablet_service.cc:1467] Processing CreateTablet for tablet 701f311c4d874d2e8c3804d32ae1340f (DEFAULT_TABLE table=test-table [id=f2245ec6ce2f46d1b922825a03bfcb17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:34.989141 25355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 701f311c4d874d2e8c3804d32ae1340f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:34.990597 25438 tablet_bootstrap.cc:492] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56: Bootstrap starting.
I20260924 10:12:34.993306 25159 heartbeater.cc:503] Master 127.6.2.254:33749 was elected leader, sending a full tablet report...
I20260924 10:12:34.994392 25438 tablet_bootstrap.cc:654] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:34.993055 25439 tablet_bootstrap.cc:492] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470: Bootstrap starting.
I20260924 10:12:34.996548 25438 log.cc:824] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:34.999198 25439 tablet_bootstrap.cc:654] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:35.000370 25439 log.cc:824] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:35.001915 25438 tablet_bootstrap.cc:492] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56: No bootstrap required, opened a new log
I20260924 10:12:35.002213 25438 ts_tablet_manager.cc:1397] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260924 10:12:35.003482 25439 tablet_bootstrap.cc:492] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470: No bootstrap required, opened a new log
I20260924 10:12:35.003758 25439 ts_tablet_manager.cc:1397] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260924 10:12:35.004019 25442 tablet_bootstrap.cc:492] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4: Bootstrap starting.
I20260924 10:12:35.008015 25442 tablet_bootstrap.cc:654] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:35.009143 25442 log.cc:824] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:35.012084 25442 tablet_bootstrap.cc:492] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4: No bootstrap required, opened a new log
I20260924 10:12:35.012372 25442 ts_tablet_manager.cc:1397] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 10:12:35.013365 25438 raft_consensus.cc:348] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71633a50f94411af879d456bdb6470" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42497 } } peers { permanent_uuid: "8e78276b476843e2bc44be8a74b3d6e4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42433 } } peers { permanent_uuid: "4471918e8a3f479a980122b773ae8e56" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41201 } }
I20260924 10:12:35.013832 25438 raft_consensus.cc:374] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:35.013940 25438 raft_consensus.cc:729] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4471918e8a3f479a980122b773ae8e56, State: Initialized, Role: FOLLOWER
I20260924 10:12:35.014607 25438 consensus_queue.cc:260] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71633a50f94411af879d456bdb6470" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42497 } } peers { permanent_uuid: "8e78276b476843e2bc44be8a74b3d6e4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42433 } } peers { permanent_uuid: "4471918e8a3f479a980122b773ae8e56" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41201 } }
I20260924 10:12:35.015823 25290 heartbeater.cc:503] Master 127.6.2.254:33749 was elected leader, sending a full tablet report...
I20260924 10:12:35.016561 25438 ts_tablet_manager.cc:1428] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260924 10:12:35.017725 25439 raft_consensus.cc:348] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71633a50f94411af879d456bdb6470" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42497 } } peers { permanent_uuid: "8e78276b476843e2bc44be8a74b3d6e4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42433 } } peers { permanent_uuid: "4471918e8a3f479a980122b773ae8e56" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41201 } }
I20260924 10:12:35.018117 25439 raft_consensus.cc:374] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:35.018239 25439 raft_consensus.cc:729] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd71633a50f94411af879d456bdb6470, State: Initialized, Role: FOLLOWER
I20260924 10:12:35.018908 25439 consensus_queue.cc:260] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71633a50f94411af879d456bdb6470" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42497 } } peers { permanent_uuid: "8e78276b476843e2bc44be8a74b3d6e4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42433 } } peers { permanent_uuid: "4471918e8a3f479a980122b773ae8e56" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41201 } }
I20260924 10:12:35.020812 25439 ts_tablet_manager.cc:1428] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470: Time spent starting tablet: real 0.017s user 0.010s sys 0.003s
I20260924 10:12:35.021386 25442 raft_consensus.cc:348] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71633a50f94411af879d456bdb6470" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42497 } } peers { permanent_uuid: "8e78276b476843e2bc44be8a74b3d6e4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42433 } } peers { permanent_uuid: "4471918e8a3f479a980122b773ae8e56" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41201 } }
I20260924 10:12:35.021829 25442 raft_consensus.cc:374] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:35.021966 25442 raft_consensus.cc:729] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e78276b476843e2bc44be8a74b3d6e4, State: Initialized, Role: FOLLOWER
I20260924 10:12:35.022769 25442 consensus_queue.cc:260] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71633a50f94411af879d456bdb6470" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42497 } } peers { permanent_uuid: "8e78276b476843e2bc44be8a74b3d6e4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42433 } } peers { permanent_uuid: "4471918e8a3f479a980122b773ae8e56" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41201 } }
I20260924 10:12:35.023916 25420 heartbeater.cc:503] Master 127.6.2.254:33749 was elected leader, sending a full tablet report...
I20260924 10:12:35.024622 25442 ts_tablet_manager.cc:1428] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260924 10:12:35.030125 25445 raft_consensus.cc:482] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:12:35.030433 25445 raft_consensus.cc:504] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71633a50f94411af879d456bdb6470" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42497 } } peers { permanent_uuid: "8e78276b476843e2bc44be8a74b3d6e4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42433 } } peers { permanent_uuid: "4471918e8a3f479a980122b773ae8e56" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41201 } }
I20260924 10:12:35.032182 25445 leader_election.cc:288] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e78276b476843e2bc44be8a74b3d6e4 (127.6.2.195:42433), 4471918e8a3f479a980122b773ae8e56 (127.6.2.194:41201)
I20260924 10:12:35.041289 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.032638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:34569 (local address 127.6.2.195:42433)
0924 10:12:35.032851 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:35.032859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:35.033740 (+ 881us) server_negotiation.cc:407] Connection header received
0924 10:12:35.033910 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:35.033917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:35.033979 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:35.034108 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:35.034729 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.035785 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.038896 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.040174 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.040304 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:35.040411 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:35.040880 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 10:12:35.041015 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:12:35.041289 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.032840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:38159 (local address 127.6.2.194:41201)
0924 10:12:35.033074 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:35.033083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:35.033652 (+ 569us) server_negotiation.cc:407] Connection header received
0924 10:12:35.033871 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:35.033875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:35.033949 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:35.034042 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:35.034599 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.035727 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.038940 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.040185 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.040304 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:35.040398 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:35.040911 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 10:12:35.041036 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:12:35.041489 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.032426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:41201 (local address 127.6.2.193:38159)
0924 10:12:35.033546 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:35.033563 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:35.033732 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:35.034069 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:35.034073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:35.034096 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:35.034452 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:35.034462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.035954 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.035958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:35.038670 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:35.038685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.040427 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.040433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:35.040640 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:35.040727 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 10:12:35.040974 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 10:12:35.041079 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":212,"thread_start_us":945,"threads_started":1}
I20260924 10:12:35.041901 25447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.032591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:42433 (local address 127.6.2.193:34569)
0924 10:12:35.033645 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:35.033660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:35.033769 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:35.034236 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:35.034239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:35.034257 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:35.034592 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:35.034599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.035937 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.035943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:35.038670 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:35.038685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.040414 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.040421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:35.040672 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:35.040728 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 10:12:35.040824 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 10:12:35.041024 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":202,"thread_start_us":505,"threads_started":1}
I20260924 10:12:35.042487 25245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701f311c4d874d2e8c3804d32ae1340f" candidate_uuid: "fd71633a50f94411af879d456bdb6470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4471918e8a3f479a980122b773ae8e56" is_pre_election: true
I20260924 10:12:35.042927 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701f311c4d874d2e8c3804d32ae1340f" candidate_uuid: "fd71633a50f94411af879d456bdb6470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e78276b476843e2bc44be8a74b3d6e4" is_pre_election: true
I20260924 10:12:35.043149 25245 raft_consensus.cc:2454] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd71633a50f94411af879d456bdb6470 in term 0.
I20260924 10:12:35.043529 25375 raft_consensus.cc:2454] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd71633a50f94411af879d456bdb6470 in term 0.
I20260924 10:12:35.043946 25049 leader_election.cc:302] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4471918e8a3f479a980122b773ae8e56, fd71633a50f94411af879d456bdb6470; no voters:
I20260924 10:12:35.044417 25445 raft_consensus.cc:2781] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:12:35.044572 25445 raft_consensus.cc:482] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:12:35.044680 25445 raft_consensus.cc:3037] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:35.047156 25445 raft_consensus.cc:504] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71633a50f94411af879d456bdb6470" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42497 } } peers { permanent_uuid: "8e78276b476843e2bc44be8a74b3d6e4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42433 } } peers { permanent_uuid: "4471918e8a3f479a980122b773ae8e56" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41201 } }
I20260924 10:12:35.047819 25445 leader_election.cc:288] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [CANDIDATE]: Term 1 election: Requested vote from peers 8e78276b476843e2bc44be8a74b3d6e4 (127.6.2.195:42433), 4471918e8a3f479a980122b773ae8e56 (127.6.2.194:41201)
I20260924 10:12:35.048280 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701f311c4d874d2e8c3804d32ae1340f" candidate_uuid: "fd71633a50f94411af879d456bdb6470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e78276b476843e2bc44be8a74b3d6e4"
I20260924 10:12:35.048391 25245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701f311c4d874d2e8c3804d32ae1340f" candidate_uuid: "fd71633a50f94411af879d456bdb6470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4471918e8a3f479a980122b773ae8e56"
I20260924 10:12:35.048514 25375 raft_consensus.cc:3037] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:35.048595 25245 raft_consensus.cc:3037] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:35.051836 25375 raft_consensus.cc:2454] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd71633a50f94411af879d456bdb6470 in term 1.
I20260924 10:12:35.051837 25245 raft_consensus.cc:2454] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd71633a50f94411af879d456bdb6470 in term 1.
I20260924 10:12:35.052301 25047 leader_election.cc:302] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4471918e8a3f479a980122b773ae8e56, 8e78276b476843e2bc44be8a74b3d6e4, fd71633a50f94411af879d456bdb6470; no voters:
I20260924 10:12:35.052649 25445 raft_consensus.cc:2781] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:35.053107 25445 raft_consensus.cc:686] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [term 1 LEADER]: Becoming Leader. State: Replica: fd71633a50f94411af879d456bdb6470, State: Running, Role: LEADER
I20260924 10:12:35.053583 25445 consensus_queue.cc:237] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71633a50f94411af879d456bdb6470" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42497 } } peers { permanent_uuid: "8e78276b476843e2bc44be8a74b3d6e4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42433 } } peers { permanent_uuid: "4471918e8a3f479a980122b773ae8e56" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41201 } }
I20260924 10:12:35.058233 24972 catalog_manager.cc:5495] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd71633a50f94411af879d456bdb6470 (127.6.2.193). New cstate: current_term: 1 leader_uuid: "fd71633a50f94411af879d456bdb6470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71633a50f94411af879d456bdb6470" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e78276b476843e2bc44be8a74b3d6e4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4471918e8a3f479a980122b773ae8e56" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41201 } health_report { overall_health: UNKNOWN } } }
I20260924 10:12:35.096665 25432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.090192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.6.2.193:42497)
0924 10:12:35.090596 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:35.090604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:35.090624 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:12:35.090754 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:35.090762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:35.090881 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:35.091089 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:12:35.091613 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.092731 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.094518 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.094928 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.095029 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:35.095644 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:12:35.096156 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:12:35.096287 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:35.096423 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 10:12:35.096462 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260924 10:12:35.105664 25375 raft_consensus.cc:1261] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4 [term 1 FOLLOWER]: Refusing update from remote peer fd71633a50f94411af879d456bdb6470: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:12:35.105664 25245 raft_consensus.cc:1261] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56 [term 1 FOLLOWER]: Refusing update from remote peer fd71633a50f94411af879d456bdb6470: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:12:35.106792 25445 consensus_queue.cc:1035] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e78276b476843e2bc44be8a74b3d6e4" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:12:35.107173 25449 consensus_queue.cc:1035] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4471918e8a3f479a980122b773ae8e56" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 41201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:12:35.123704 25452 mvcc.cc:205] Tried to move back new op lower bound from 7332842516901183488 to 7332842516703072256. Current Snapshot: MvccSnapshot[applied={T|T < 7332842516901183488 or (T in {7332842516901183488})}]
I20260924 10:12:35.132817 24972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:12:35.133141 24972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:12:35.137276 24972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcde56466a9d472c8c09e0dea1b746c9: Sending DeleteTablet for 3 replicas of tablet 701f311c4d874d2e8c3804d32ae1340f
I20260924 10:12:35.138562 25094 tablet_service.cc:1514] Processing DeleteTablet for tablet 701f311c4d874d2e8c3804d32ae1340f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:35 UTC) from {username='slave'} at 127.0.0.1:41114
I20260924 10:12:35.138622 25355 tablet_service.cc:1514] Processing DeleteTablet for tablet 701f311c4d874d2e8c3804d32ae1340f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:35 UTC) from {username='slave'} at 127.0.0.1:59402
I20260924 10:12:35.138722 25225 tablet_service.cc:1514] Processing DeleteTablet for tablet 701f311c4d874d2e8c3804d32ae1340f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:35 UTC) from {username='slave'} at 127.0.0.1:38862
I20260924 10:12:35.141474 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 25032
I20260924 10:12:35.151242 25462 tablet_replica.cc:330] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470: stopping tablet replica
I20260924 10:12:35.151862 25462 raft_consensus.cc:2229] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:12:35.152725 25462 raft_consensus.cc:2258] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:35.152870 25463 tablet_replica.cc:330] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56: stopping tablet replica
I20260924 10:12:35.153299 25463 raft_consensus.cc:2229] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:12:35.153812 25463 raft_consensus.cc:2258] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:35.156199 25463 ts_tablet_manager.cc:1905] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:35.157323 25461 tablet_replica.cc:330] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4: stopping tablet replica
I20260924 10:12:35.157743 25461 raft_consensus.cc:2229] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:12:35.158243 25461 raft_consensus.cc:2258] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:35.160002 25462 ts_tablet_manager.cc:1905] T 701f311c4d874d2e8c3804d32ae1340f P fd71633a50f94411af879d456bdb6470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:35.163576 25463 ts_tablet_manager.cc:1918] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:35.163734 25463 log.cc:1196] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/701f311c4d874d2e8c3804d32ae1340f
I20260924 10:12:35.163940 25432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.143224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.6.2.193:42497)
0924 10:12:35.143471 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:35.143478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:35.154786 (+ 11308us) server_negotiation.cc:407] Connection header received
0924 10:12:35.154975 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:35.154983 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:35.155133 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:35.155261 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:35.155944 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.157055 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.159743 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.160402 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.160519 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:35.162089 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:35.162100 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:35.162121 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:35.162931 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:35.162989 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:35.163013 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:35.163133 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:35.163406 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 10:12:35.163516 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 10:12:35.164294 25463 ts_tablet_manager.cc:1939] T 701f311c4d874d2e8c3804d32ae1340f P 4471918e8a3f479a980122b773ae8e56: Deleting consensus metadata
I20260924 10:12:35.166129 25154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:35.167222 25461 ts_tablet_manager.cc:1905] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:35.168337 24959 catalog_manager.cc:4841] TS 4471918e8a3f479a980122b773ae8e56 (127.6.2.194:41201): tablet 701f311c4d874d2e8c3804d32ae1340f (table test-table [id=f2245ec6ce2f46d1b922825a03bfcb17]) successfully deleted
I20260924 10:12:35.173501 25461 ts_tablet_manager.cc:1918] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:35.173703 25461 log.cc:1196] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/701f311c4d874d2e8c3804d32ae1340f
I20260924 10:12:35.174137 25461 ts_tablet_manager.cc:1939] T 701f311c4d874d2e8c3804d32ae1340f P 8e78276b476843e2bc44be8a74b3d6e4: Deleting consensus metadata
I20260924 10:12:35.175896 24957 catalog_manager.cc:4841] TS 8e78276b476843e2bc44be8a74b3d6e4 (127.6.2.195:42433): tablet 701f311c4d874d2e8c3804d32ae1340f (table test-table [id=f2245ec6ce2f46d1b922825a03bfcb17]) successfully deleted
I20260924 10:12:35.265981 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 25032
W20260924 10:12:35.284596 24957 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.2.193:42497 (error 108)
I20260924 10:12:35.284855 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 25163
I20260924 10:12:35.286613 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.286096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42497 (local address 127.0.0.1:41140)
0924 10:12:35.286257 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:35.286409 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 10:12:35.286865 24957 catalog_manager.cc:4584] TS fd71633a50f94411af879d456bdb6470 (127.6.2.193:42497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 701f311c4d874d2e8c3804d32ae1340f: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42497: connect: Connection refused (error 111)
I20260924 10:12:35.291883 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.285756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.6.2.194:41201)
0924 10:12:35.285942 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:35.285951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:35.285973 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:12:35.286129 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:35.286135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:35.286238 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:35.286346 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:35.286845 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.287706 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.288992 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.289396 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.289502 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:35.290427 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:35.290435 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:35.290449 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:35.291031 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:35.291262 (+ 231us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:35.291287 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:35.291380 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:35.291529 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 10:12:35.291618 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:12:35.292621 25285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:35.340444 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.339962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42497 (local address 127.0.0.1:41142)
0924 10:12:35.340165 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:35.340290 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 10:12:35.341506 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.341122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42497 (local address 127.0.0.1:41154)
0924 10:12:35.341298 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:35.341372 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 10:12:35.383765 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 25163
I20260924 10:12:35.399848 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 25293
I20260924 10:12:35.401729 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.401333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42497 (local address 127.0.0.1:41162)
0924 10:12:35.401472 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:35.401549 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 10:12:35.402784 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.402373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42497 (local address 127.0.0.1:41174)
0924 10:12:35.402533 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:35.402621 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:12:35.406921 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.400629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.6.2.195:42433)
0924 10:12:35.400841 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:35.400847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:35.400888 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:12:35.401071 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:35.401076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:35.401183 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:35.401269 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:35.401713 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.402565 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.403954 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:35.404349 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:35.404447 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:35.405452 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:35.405461 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:35.405476 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:35.406125 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:35.406173 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:35.406192 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:35.406310 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:35.406475 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 10:12:35.406606 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 10:12:35.407675 25415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:35.483582 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.483122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42497 (local address 127.0.0.1:41186)
0924 10:12:35.483323 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:35.483421 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 10:12:35.484694 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:35.484257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42497 (local address 127.0.0.1:41188)
0924 10:12:35.484477 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:35.484570 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 10:12:35.498066 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 25293
I20260924 10:12:35.515201 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 24942
I20260924 10:12:35.516319 25003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:35.629056 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 24942
2026-09-24T10:12:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2764 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:12:35 dist-test-slave-s3n3 krb5kdc[25472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:12:35 dist-test-slave-s3n3 krb5kdc[25472](info): set up 2 sockets
Sep 24 10:12:35 dist-test-slave-s3n3 krb5kdc[25472](info): commencing operation
krb5kdc: starting...
W20260924 10:12:36.743625 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:12:36 dist-test-slave-s3n3 krb5kdc[25472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:12:36.999697 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:39085
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:39085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:37.336614 25485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:37.336915 25485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:37.337090 25485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:37.344502 25485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:37.344620 25485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:37.344710 25485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:37.344822 25485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:37.344900 25485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:37.354944 25485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:39085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:39085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:37.356360 25485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:37.357771 25485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:37.369663 25490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:37.369724 25493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:37.369807 25491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:37.370541 25485 server_base.cc:1034] running on GCE node
I20260924 10:12:37.371344 25485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:37.372717 25485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:37.373920 25485 hybrid_clock.cc:648] HybridClock initialized: now 1790244757373862 us; error 49 us; skew 500 ppm
I20260924 10:12:37.374219 25485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:37 dist-test-slave-s3n3 krb5kdc[25472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244757, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:37.377351 25485 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:12:37.378938 25485 webserver.cc:460] Webserver started at http://127.6.2.254:39883/ using document root <none> and password file <none>
I20260924 10:12:37.379670 25485 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:37.379798 25485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:37.380119 25485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:37.383211 25485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "495e791597ec48cab0283552b3ebeca9"
format_stamp: "Formatted at 2026-09-24 10:12:37 on dist-test-slave-s3n3"
server_key: "229b8a39a24e5d769e1582cacf6305d6"
server_key_iv: "7faf493ad743248532208f9dbc959ace"
server_key_version: "encryptionkey@0"
I20260924 10:12:37.383980 25485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "495e791597ec48cab0283552b3ebeca9"
format_stamp: "Formatted at 2026-09-24 10:12:37 on dist-test-slave-s3n3"
server_key: "229b8a39a24e5d769e1582cacf6305d6"
server_key_iv: "7faf493ad743248532208f9dbc959ace"
server_key_version: "encryptionkey@0"
I20260924 10:12:37.389384 25485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:12:37.393169 25500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:37.394625 25485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:12:37.394826 25485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "495e791597ec48cab0283552b3ebeca9"
format_stamp: "Formatted at 2026-09-24 10:12:37 on dist-test-slave-s3n3"
server_key: "229b8a39a24e5d769e1582cacf6305d6"
server_key_iv: "7faf493ad743248532208f9dbc959ace"
server_key_version: "encryptionkey@0"
I20260924 10:12:37.395041 25485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:37.417335 25485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:37.421375 25485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:37.421670 25485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:37.440763 25485 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:39085
I20260924 10:12:37.440812 25551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:39085 every 8 connection(s)
I20260924 10:12:37.442173 25485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:37.444875 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 25485
I20260924 10:12:37.445070 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:37.445608 6155 external_mini_cluster.cc:1428] Setting key 08b1a0138864775cb43fa8e0e5492ffc
I20260924 10:12:37.446561 25552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:37.458927 25552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9: Bootstrap starting.
Sep 24 10:12:37 dist-test-slave-s3n3 krb5kdc[25472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:12:37.462456 25552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:37.463629 25552 log.cc:824] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:37.467118 25552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9: No bootstrap required, opened a new log
I20260924 10:12:37.468333 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:37.447449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.6.2.254:39085)
0924 10:12:37.448198 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:37.448209 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:37.448248 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:12:37.449912 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:37.449933 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:37.450307 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:37.450746 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:37.452251 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:37.453570 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:37.454853 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:37.455578 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:37.455809 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:37.460132 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:37.460140 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:37.460154 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:37.460193 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:37.463532 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:37.464357 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:37.464367 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:37.464373 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:37.464541 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:37.465013 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:37.465017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:37.465021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:37.465780 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:37.466094 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:37.466519 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 10:12:37.466972 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":355,"threads_started":1}
I20260924 10:12:37.475838 25552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495e791597ec48cab0283552b3ebeca9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39085 } }
I20260924 10:12:37.476297 25552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:37.476454 25552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495e791597ec48cab0283552b3ebeca9, State: Initialized, Role: FOLLOWER
I20260924 10:12:37.477021 25552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495e791597ec48cab0283552b3ebeca9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39085 } }
I20260924 10:12:37.477232 25552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:37.477334 25552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:37.477456 25552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:37.479681 25552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495e791597ec48cab0283552b3ebeca9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39085 } }
I20260924 10:12:37.480124 25552 leader_election.cc:302] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 495e791597ec48cab0283552b3ebeca9; no voters:
I20260924 10:12:37.480679 25552 leader_election.cc:288] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:37.480837 25557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:37.481576 25557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [term 1 LEADER]: Becoming Leader. State: Replica: 495e791597ec48cab0283552b3ebeca9, State: Running, Role: LEADER
I20260924 10:12:37.482282 25557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495e791597ec48cab0283552b3ebeca9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39085 } }
I20260924 10:12:37.482909 25552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:37.485466 25559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 495e791597ec48cab0283552b3ebeca9. Latest consensus state: current_term: 1 leader_uuid: "495e791597ec48cab0283552b3ebeca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495e791597ec48cab0283552b3ebeca9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39085 } } }
I20260924 10:12:37.485440 25558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "495e791597ec48cab0283552b3ebeca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495e791597ec48cab0283552b3ebeca9" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39085 } } }
I20260924 10:12:37.485917 25559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:37.485908 25558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:37.488441 25564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:37.495131 25564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:37.505226 25564 catalog_manager.cc:1347] Generated new cluster ID: 7bbbe92d3050468ab3a8a976e8da7b04
I20260924 10:12:37.505365 25564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:37.525928 25564 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:37.527393 25564 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:37.539433 25564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9: Generated new TSK 0
I20260924 10:12:37.540328 25564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.2.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
Entry for principal kudu/127.6.2.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab.
I20260924 10:12:37.603163 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:39085
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:37.940642 25578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:37.940977 25578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:37.941151 25578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:37.948383 25578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:37.948547 25578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:37.948738 25578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:12:37.958632 25578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.193.keytab
--principal=kudu/127.6.2.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:39085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:37.960058 25578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:37.961366 25578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:37.975848 25586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:37.975935 25584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:37.975934 25583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:37.977294 25578 server_base.cc:1034] running on GCE node
I20260924 10:12:37.977972 25578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:37.978938 25578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:37.980159 25578 hybrid_clock.cc:648] HybridClock initialized: now 1790244757980115 us; error 82 us; skew 500 ppm
I20260924 10:12:37.980487 25578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:37 dist-test-slave-s3n3 krb5kdc[25472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244757, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:37.983789 25578 init.cc:377] Logged in from keytab as kudu/127.6.2.193@KRBTEST.COM (short username kudu)
I20260924 10:12:37.985405 25578 webserver.cc:460] Webserver started at http://127.6.2.193:45375/ using document root <none> and password file <none>
I20260924 10:12:37.986076 25578 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:37.986173 25578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:37.986428 25578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:37.989362 25578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "e4deea200ad84fc48076b6dfe4682082"
format_stamp: "Formatted at 2026-09-24 10:12:37 on dist-test-slave-s3n3"
server_key: "e9dfc2ce07f2d5162e7ef9d4edaed6ad"
server_key_iv: "7b7d8864ba5e5db9d4c63ee5e64e2442"
server_key_version: "encryptionkey@0"
I20260924 10:12:37.990164 25578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "e4deea200ad84fc48076b6dfe4682082"
format_stamp: "Formatted at 2026-09-24 10:12:37 on dist-test-slave-s3n3"
server_key: "e9dfc2ce07f2d5162e7ef9d4edaed6ad"
server_key_iv: "7b7d8864ba5e5db9d4c63ee5e64e2442"
server_key_version: "encryptionkey@0"
I20260924 10:12:37.995347 25578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:12:37.998878 25593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:38.000303 25578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:12:38.000469 25578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "e4deea200ad84fc48076b6dfe4682082"
format_stamp: "Formatted at 2026-09-24 10:12:37 on dist-test-slave-s3n3"
server_key: "e9dfc2ce07f2d5162e7ef9d4edaed6ad"
server_key_iv: "7b7d8864ba5e5db9d4c63ee5e64e2442"
server_key_version: "encryptionkey@0"
I20260924 10:12:38.000681 25578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:38.014902 25578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:38.019500 25578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:38.019829 25578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:38.020854 25578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:38.022238 25578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:38.022312 25578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:38.022464 25578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:38.022533 25578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:38.056896 25578 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:46407
I20260924 10:12:38.057001 25705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:46407 every 8 connection(s)
I20260924 10:12:38.058264 25578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:12:38.061487 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 25578
I20260924 10:12:38.061702 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:12:38.062295 6155 external_mini_cluster.cc:1428] Setting key c3f5e8e42dd8ff3c0454d3fec784fc87
Sep 24 10:12:38 dist-test-slave-s3n3 krb5kdc[25472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244757, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.193@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:12:38.079483 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:38.060873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:41899 (local address 127.6.2.254:39085)
0924 10:12:38.061400 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:38.061405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:38.062353 (+ 948us) server_negotiation.cc:407] Connection header received
0924 10:12:38.064835 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:38.064840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:38.064937 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:38.065054 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:38.066542 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:38.067327 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:38.068896 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:38.069294 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:38.069446 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:38.073440 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:38.073445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:38.073449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:38.073482 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:38.076108 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:38.077070 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:38.077074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:38.077077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:38.077186 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:38.077671 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:38.077674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:38.077676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:38.077951 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:38.078063 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:38.078960 (+ 897us) server_negotiation.cc:299] Negotiation successful
0924 10:12:38.079200 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":293,"threads_started":1}
I20260924 10:12:38.081354 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:38.061305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:39085 (local address 127.6.2.193:41899)
0924 10:12:38.062186 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:38.062230 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:38.064556 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:38.065289 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:38.065297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:38.065802 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:38.066373 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:38.066390 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:38.067475 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:38.067485 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:38.068724 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:38.068736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:38.069544 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:38.069554 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:38.069813 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:38.070532 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:12:38.070555 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:38.073167 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:38.076267 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:38.076278 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:38.076299 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:38.076955 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:38.077301 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:38.077308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:38.077314 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:38.077563 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:38.078087 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:38.078098 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:38.078554 (+ 456us) client_negotiation.cc:769] Sending connection context
0924 10:12:38.078856 (+ 302us) client_negotiation.cc:240] Negotiation successful
0924 10:12:38.079474 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":408,"threads_started":1}
I20260924 10:12:38.083161 25706 heartbeater.cc:347] Connected to a master server at 127.6.2.254:39085
I20260924 10:12:38.083554 25706 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:38.084357 25706 heartbeater.cc:511] Master 127.6.2.254:39085 requested a full tablet report, sending...
I20260924 10:12:38.086602 25517 ts_manager.cc:194] Registered new tserver with Master: e4deea200ad84fc48076b6dfe4682082 (127.6.2.193:46407)
I20260924 10:12:38.088121 25517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.193@KRBTEST.COM'} at 127.6.2.193:41899
WARNING: no policy specified for kudu/127.6.2.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
Entry for principal kudu/127.6.2.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab.
I20260924 10:12:38.122807 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:39085
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:38.456938 25712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:38.457259 25712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:38.457429 25712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:38.465008 25712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:38.465165 25712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:38.465345 25712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:12:38.475333 25712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.194.keytab
--principal=kudu/127.6.2.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:39085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:38.476744 25712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:38.478065 25712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:38.491134 25717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:38.491104 25718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:38.491168 25720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:38.491937 25712 server_base.cc:1034] running on GCE node
I20260924 10:12:38.492547 25712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:38.493508 25712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:38.494751 25712 hybrid_clock.cc:648] HybridClock initialized: now 1790244758494693 us; error 45 us; skew 500 ppm
I20260924 10:12:38.495168 25712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:38 dist-test-slave-s3n3 krb5kdc[25472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244758, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:38.498414 25712 init.cc:377] Logged in from keytab as kudu/127.6.2.194@KRBTEST.COM (short username kudu)
I20260924 10:12:38.500237 25712 webserver.cc:460] Webserver started at http://127.6.2.194:46045/ using document root <none> and password file <none>
I20260924 10:12:38.500975 25712 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:38.501118 25712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:38.501444 25712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:38.504455 25712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "e3b43cf9d7a6451488d805a0f4894482"
format_stamp: "Formatted at 2026-09-24 10:12:38 on dist-test-slave-s3n3"
server_key: "13999e3edd9073c8f1ef11f4b0a41739"
server_key_iv: "56096bca4fe16b1a1396310e9012eed2"
server_key_version: "encryptionkey@0"
I20260924 10:12:38.505302 25712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "e3b43cf9d7a6451488d805a0f4894482"
format_stamp: "Formatted at 2026-09-24 10:12:38 on dist-test-slave-s3n3"
server_key: "13999e3edd9073c8f1ef11f4b0a41739"
server_key_iv: "56096bca4fe16b1a1396310e9012eed2"
server_key_version: "encryptionkey@0"
I20260924 10:12:38.510432 25712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 10:12:38.513989 25727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:38.515398 25712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:12:38.515612 25712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "e3b43cf9d7a6451488d805a0f4894482"
format_stamp: "Formatted at 2026-09-24 10:12:38 on dist-test-slave-s3n3"
server_key: "13999e3edd9073c8f1ef11f4b0a41739"
server_key_iv: "56096bca4fe16b1a1396310e9012eed2"
server_key_version: "encryptionkey@0"
I20260924 10:12:38.515900 25712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:38.536849 25712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:38.541255 25712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:38.541508 25712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:38.542418 25712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:38.543723 25712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:38.543798 25712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:38.543887 25712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:38.543933 25712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:38.576705 25712 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:33029
I20260924 10:12:38.576756 25839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:33029 every 8 connection(s)
I20260924 10:12:38.578004 25712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:12:38.578464 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 25712
I20260924 10:12:38.578653 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:12:38.579306 6155 external_mini_cluster.cc:1428] Setting key 39b3b414f7ba59e2dbc53bde9a8e3d13
Sep 24 10:12:38 dist-test-slave-s3n3 krb5kdc[25472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244758, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.194@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.2.195@KRBTEST.COM; defaulting to no policy
I20260924 10:12:38.606966 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:38.581259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:39377 (local address 127.6.2.254:39085)
0924 10:12:38.581769 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:38.581788 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:38.582835 (+ 1047us) server_negotiation.cc:407] Connection header received
0924 10:12:38.584872 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:38.584878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:38.584969 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:38.585091 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:38.586596 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:38.587412 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:38.589014 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:38.589448 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:38.589585 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:38.599721 (+ 10136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:38.599731 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:38.599739 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:38.599800 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:38.603552 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:38.604395 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:38.604402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:38.604420 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:38.604580 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:38.605196 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:38.605203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:38.605211 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:38.605499 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:38.605611 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:38.606429 (+ 818us) server_negotiation.cc:299] Negotiation successful
0924 10:12:38.606608 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":278,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.195@KRBTEST.COM" created.
I20260924 10:12:38.608057 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:38.581854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:39085 (local address 127.6.2.194:39377)
0924 10:12:38.582637 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:38.582688 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:38.584618 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:38.585325 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:38.585333 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:38.585819 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:38.586425 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:38.586443 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:38.587573 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:38.587578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:38.588866 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:38.588879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:38.589675 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:38.589684 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:38.589919 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:38.596619 (+ 6700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:12:38.596652 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:38.599364 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:38.603768 (+ 4404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:38.603776 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:38.603792 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:38.604283 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:38.604799 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:38.604806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:38.604810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:38.605074 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:38.605654 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:38.605660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:38.606095 (+ 435us) client_negotiation.cc:769] Sending connection context
0924 10:12:38.606353 (+ 258us) client_negotiation.cc:240] Negotiation successful
0924 10:12:38.606779 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":307,"threads_started":1}
I20260924 10:12:38.609639 25840 heartbeater.cc:347] Connected to a master server at 127.6.2.254:39085
I20260924 10:12:38.609972 25840 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:38.610819 25840 heartbeater.cc:511] Master 127.6.2.254:39085 requested a full tablet report, sending...
I20260924 10:12:38.612475 25517 ts_manager.cc:194] Registered new tserver with Master: e3b43cf9d7a6451488d805a0f4894482 (127.6.2.194:33029)
I20260924 10:12:38.613324 25517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.194@KRBTEST.COM'} at 127.6.2.194:39377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
Entry for principal kudu/127.6.2.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab.
I20260924 10:12:38.639166 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:39085
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:38.981382 25846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:38.981667 25846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:38.981773 25846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:38.989400 25846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:38.989526 25846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:38.989638 25846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:12:39.000030 25846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.195.keytab
--principal=kudu/127.6.2.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:39085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:39.001330 25846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:39.002676 25846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:39.016227 25852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:39.016227 25851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:39.016745 25854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:39.017179 25846 server_base.cc:1034] running on GCE node
I20260924 10:12:39.017822 25846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:39.018853 25846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:39.020077 25846 hybrid_clock.cc:648] HybridClock initialized: now 1790244759020016 us; error 55 us; skew 500 ppm
I20260924 10:12:39.020505 25846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:39 dist-test-slave-s3n3 krb5kdc[25472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244759, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:39.030095 25846 init.cc:377] Logged in from keytab as kudu/127.6.2.195@KRBTEST.COM (short username kudu)
I20260924 10:12:39.031708 25846 webserver.cc:460] Webserver started at http://127.6.2.195:35795/ using document root <none> and password file <none>
I20260924 10:12:39.032413 25846 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:39.032557 25846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:39.032888 25846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:39.035915 25846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "bf9283a7092942508761244446c78155"
format_stamp: "Formatted at 2026-09-24 10:12:39 on dist-test-slave-s3n3"
server_key: "83e4bc9969cd38338dd3b3941f2e9a49"
server_key_iv: "5473036263f876db73a04ab53eac512e"
server_key_version: "encryptionkey@0"
I20260924 10:12:39.036638 25846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "bf9283a7092942508761244446c78155"
format_stamp: "Formatted at 2026-09-24 10:12:39 on dist-test-slave-s3n3"
server_key: "83e4bc9969cd38338dd3b3941f2e9a49"
server_key_iv: "5473036263f876db73a04ab53eac512e"
server_key_version: "encryptionkey@0"
I20260924 10:12:39.041805 25846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:12:39.045207 25861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:39.046482 25846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:12:39.046615 25846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "bf9283a7092942508761244446c78155"
format_stamp: "Formatted at 2026-09-24 10:12:39 on dist-test-slave-s3n3"
server_key: "83e4bc9969cd38338dd3b3941f2e9a49"
server_key_iv: "5473036263f876db73a04ab53eac512e"
server_key_version: "encryptionkey@0"
I20260924 10:12:39.046774 25846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:39.072227 25846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:39.076232 25846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:39.076488 25846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:39.077390 25846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:39.078716 25846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:39.078789 25846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:39.078871 25846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:39.078914 25846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:39.091185 25706 heartbeater.cc:503] Master 127.6.2.254:39085 was elected leader, sending a full tablet report...
I20260924 10:12:39.113893 25846 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:32855
I20260924 10:12:39.113940 25973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:32855 every 8 connection(s)
I20260924 10:12:39.115260 25846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:12:39.124794 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 25846
I20260924 10:12:39.124995 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:12:39.125440 6155 external_mini_cluster.cc:1428] Setting key a9ce96b343e71219a7f999be3504b063
Sep 24 10:12:39 dist-test-slave-s3n3 krb5kdc[25472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244759, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.195@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:12:39.135507 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.117798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:59979 (local address 127.6.2.254:39085)
0924 10:12:39.118254 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:39.118258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:39.119133 (+ 875us) server_negotiation.cc:407] Connection header received
0924 10:12:39.121157 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:39.121163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:39.121270 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:39.121402 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:39.122877 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.123737 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.125409 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.125846 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.125986 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.129852 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:39.129857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:39.129860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:39.129929 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:39.132472 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:39.133186 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:39.133190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:39.133192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:39.133327 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:39.133685 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:39.133688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:39.133690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:39.133953 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:39.134074 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:39.134853 (+ 779us) server_negotiation.cc:299] Negotiation successful
0924 10:12:39.135204 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":250,"threads_started":1}
I20260924 10:12:39.136534 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.118230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:39085 (local address 127.6.2.195:59979)
0924 10:12:39.118958 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:39.118995 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:39.120895 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:39.121628 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:39.121635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:39.122139 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:39.122690 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:39.122706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.123892 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.123899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:39.125224 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:39.125236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.126076 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.126083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:39.126354 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.127167 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:12:39.127189 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:39.129656 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:39.132609 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:39.132616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:39.132629 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:39.133078 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:39.133420 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:12:39.133423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:12:39.133425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:12:39.133594 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:12:39.134095 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:39.134102 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:39.134452 (+ 350us) client_negotiation.cc:769] Sending connection context
0924 10:12:39.134736 (+ 284us) client_negotiation.cc:240] Negotiation successful
0924 10:12:39.135328 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":314,"threads_started":1}
I20260924 10:12:39.138001 25974 heartbeater.cc:347] Connected to a master server at 127.6.2.254:39085
I20260924 10:12:39.138315 25974 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:39.139182 25974 heartbeater.cc:511] Master 127.6.2.254:39085 requested a full tablet report, sending...
I20260924 10:12:39.141009 25516 ts_manager.cc:194] Registered new tserver with Master: bf9283a7092942508761244446c78155 (127.6.2.195:32855)
I20260924 10:12:39.141860 25516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.2.195@KRBTEST.COM'} at 127.6.2.195:59979
I20260924 10:12:39.153234 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:12:39.173913 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.157487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.6.2.254:39085)
0924 10:12:39.157650 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:39.157655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:39.158159 (+ 504us) server_negotiation.cc:407] Connection header received
0924 10:12:39.158379 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:39.158384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:39.158449 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:39.158553 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:39.159956 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.160694 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.162056 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.162471 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.162603 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.164942 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:39.164949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:39.164955 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:39.164993 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:39.167548 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:39.168419 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:39.168429 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:39.168432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:39.168550 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:39.169319 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:39.169342 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:39.169346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:39.169814 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:39.170071 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:39.173019 (+ 2948us) server_negotiation.cc:299] Negotiation successful
0924 10:12:39.173329 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:12:39.179292 25516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:12:39.189921 25516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:12:39.224227 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.203586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:33029 (local address 127.0.0.1:44092)
0924 10:12:39.213641 (+ 10055us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:39.213697 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:39.213974 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:39.216037 (+ 2063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:39.216044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:39.216070 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:39.216512 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:39.216526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.218755 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.218761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:39.221739 (+ 2978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:39.221753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.223117 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.223125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:39.223322 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.223385 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 10:12:39.223539 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 10:12:39.223743 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9776,"mutex_wait_us":117,"thread_start_us":11654,"threads_started":1}
I20260924 10:12:39.224462 25992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.205009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.6.2.193:46407)
0924 10:12:39.213627 (+ 8618us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:39.213636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:39.215762 (+ 2126us) server_negotiation.cc:407] Connection header received
0924 10:12:39.215909 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:39.215916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:39.216136 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:39.216325 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:39.217336 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.218859 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.221919 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.223383 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.223515 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.223741 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:39.223936 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 10:12:39.224133 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8443,"thread_start_us":8278,"threads_started":1}
I20260924 10:12:39.225068 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.204417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.6.2.194:33029)
0924 10:12:39.215085 (+ 10668us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:39.215095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:39.215128 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:12:39.215245 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:39.215255 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:39.215548 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:39.215739 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:39.216849 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.218596 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.221935 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.222850 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.223013 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.223258 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:39.224710 (+ 1452us) server_negotiation.cc:299] Negotiation successful
0924 10:12:39.224842 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10499,"thread_start_us":10390,"threads_started":1}
I20260924 10:12:39.224445 25987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.204208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:46407 (local address 127.0.0.1:59604)
0924 10:12:39.215500 (+ 11292us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:39.215519 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:39.215635 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:39.216592 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:39.216598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:39.216627 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:39.217104 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:39.217118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.219037 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.219045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:39.221741 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:39.221753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.223595 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.223601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:39.223754 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.223784 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:12:39.223894 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 10:12:39.224013 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11160,"mutex_wait_us":191,"thread_start_us":8988,"threads_started":1}
I20260924 10:12:39.225559 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.204245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.6.2.195:32855)
0924 10:12:39.216141 (+ 11896us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:39.216149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:39.216183 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:12:39.216272 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:39.216278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:39.216483 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:39.216698 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:39.217477 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.218537 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.221255 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.223635 (+ 2380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.223774 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.223975 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:39.225049 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0924 10:12:39.225203 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11720,"thread_start_us":12697,"threads_started":1}
I20260924 10:12:39.224953 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.203985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:32855 (local address 127.0.0.1:41896)
0924 10:12:39.214350 (+ 10365us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:39.214374 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:39.214537 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:39.216701 (+ 2164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:39.216706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:39.216754 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:39.217300 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:39.217311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.218742 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.218747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:39.221099 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:39.221112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.224131 (+ 3019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.224137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:39.224338 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.224370 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:12:39.224474 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 10:12:39.224611 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10195,"mutex_wait_us":103,"spinlock_wait_cycles":10496,"thread_start_us":10047,"threads_started":1}
I20260924 10:12:39.229364 25909 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4909efb514432dab83e02fbf990ec1 (DEFAULT_TABLE table=test-table [id=d352691869d64115947a4a3187bc1e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:39.229368 25775 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4909efb514432dab83e02fbf990ec1 (DEFAULT_TABLE table=test-table [id=d352691869d64115947a4a3187bc1e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:39.230878 25775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4909efb514432dab83e02fbf990ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:39.230891 25909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4909efb514432dab83e02fbf990ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:39.232596 25641 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4909efb514432dab83e02fbf990ec1 (DEFAULT_TABLE table=test-table [id=d352691869d64115947a4a3187bc1e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:39.233727 25641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4909efb514432dab83e02fbf990ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:39.245692 25993 tablet_bootstrap.cc:492] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082: Bootstrap starting.
I20260924 10:12:39.246538 25994 tablet_bootstrap.cc:492] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482: Bootstrap starting.
I20260924 10:12:39.247349 25995 tablet_bootstrap.cc:492] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155: Bootstrap starting.
I20260924 10:12:39.249668 25993 tablet_bootstrap.cc:654] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:39.250780 25995 tablet_bootstrap.cc:654] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:39.251035 25993 log.cc:824] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:39.251995 25995 log.cc:824] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:39.254323 25994 tablet_bootstrap.cc:654] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:39.254879 25993 tablet_bootstrap.cc:492] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082: No bootstrap required, opened a new log
I20260924 10:12:39.255244 25993 ts_tablet_manager.cc:1397] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260924 10:12:39.255535 25994 log.cc:824] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:39.257781 25995 tablet_bootstrap.cc:492] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155: No bootstrap required, opened a new log
I20260924 10:12:39.258038 25995 ts_tablet_manager.cc:1397] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260924 10:12:39.266149 25993 raft_consensus.cc:348] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b43cf9d7a6451488d805a0f4894482" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 33029 } } peers { permanent_uuid: "bf9283a7092942508761244446c78155" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 32855 } } peers { permanent_uuid: "e4deea200ad84fc48076b6dfe4682082" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 46407 } }
I20260924 10:12:39.266623 25993 raft_consensus.cc:374] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:39.266721 25993 raft_consensus.cc:729] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4deea200ad84fc48076b6dfe4682082, State: Initialized, Role: FOLLOWER
I20260924 10:12:39.267417 25993 consensus_queue.cc:260] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b43cf9d7a6451488d805a0f4894482" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 33029 } } peers { permanent_uuid: "bf9283a7092942508761244446c78155" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 32855 } } peers { permanent_uuid: "e4deea200ad84fc48076b6dfe4682082" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 46407 } }
I20260924 10:12:39.268182 25994 tablet_bootstrap.cc:492] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482: No bootstrap required, opened a new log
I20260924 10:12:39.268491 25994 ts_tablet_manager.cc:1397] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.010s
I20260924 10:12:39.268558 25995 raft_consensus.cc:348] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b43cf9d7a6451488d805a0f4894482" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 33029 } } peers { permanent_uuid: "bf9283a7092942508761244446c78155" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 32855 } } peers { permanent_uuid: "e4deea200ad84fc48076b6dfe4682082" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 46407 } }
I20260924 10:12:39.268990 25995 raft_consensus.cc:374] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:39.269106 25995 raft_consensus.cc:729] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9283a7092942508761244446c78155, State: Initialized, Role: FOLLOWER
I20260924 10:12:39.269266 25993 ts_tablet_manager.cc:1428] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260924 10:12:39.269775 25995 consensus_queue.cc:260] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b43cf9d7a6451488d805a0f4894482" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 33029 } } peers { permanent_uuid: "bf9283a7092942508761244446c78155" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 32855 } } peers { permanent_uuid: "e4deea200ad84fc48076b6dfe4682082" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 46407 } }
I20260924 10:12:39.270857 25974 heartbeater.cc:503] Master 127.6.2.254:39085 was elected leader, sending a full tablet report...
I20260924 10:12:39.271483 25995 ts_tablet_manager.cc:1428] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155: Time spent starting tablet: real 0.013s user 0.009s sys 0.006s
I20260924 10:12:39.278020 25994 raft_consensus.cc:348] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b43cf9d7a6451488d805a0f4894482" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 33029 } } peers { permanent_uuid: "bf9283a7092942508761244446c78155" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 32855 } } peers { permanent_uuid: "e4deea200ad84fc48076b6dfe4682082" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 46407 } }
I20260924 10:12:39.278491 25994 raft_consensus.cc:374] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:39.278605 25994 raft_consensus.cc:729] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b43cf9d7a6451488d805a0f4894482, State: Initialized, Role: FOLLOWER
I20260924 10:12:39.279196 25994 consensus_queue.cc:260] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b43cf9d7a6451488d805a0f4894482" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 33029 } } peers { permanent_uuid: "bf9283a7092942508761244446c78155" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 32855 } } peers { permanent_uuid: "e4deea200ad84fc48076b6dfe4682082" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 46407 } }
I20260924 10:12:39.280195 25840 heartbeater.cc:503] Master 127.6.2.254:39085 was elected leader, sending a full tablet report...
I20260924 10:12:39.280843 25994 ts_tablet_manager.cc:1428] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
W20260924 10:12:39.310782 25707 tablet.cc:2369] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:12:39.330173 25841 tablet.cc:2369] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:12:39.358279 26000 raft_consensus.cc:482] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:12:39.358583 26000 raft_consensus.cc:504] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b43cf9d7a6451488d805a0f4894482" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 33029 } } peers { permanent_uuid: "bf9283a7092942508761244446c78155" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 32855 } } peers { permanent_uuid: "e4deea200ad84fc48076b6dfe4682082" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 46407 } }
I20260924 10:12:39.360438 26000 leader_election.cc:288] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3b43cf9d7a6451488d805a0f4894482 (127.6.2.194:33029), e4deea200ad84fc48076b6dfe4682082 (127.6.2.193:46407)
W20260924 10:12:39.366933 25975 tablet.cc:2369] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:12:39.367480 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.360773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:46407 (local address 127.6.2.195:57779)
0924 10:12:39.361038 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:39.361067 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:39.361264 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:39.361725 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:39.361730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:39.361753 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:39.362026 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:39.362039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.363330 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.363336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:39.365062 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:39.365075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.366494 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.366501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:39.366750 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.366792 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 10:12:39.366906 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 10:12:39.367094 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":1,"spinlock_wait_cycles":28416}
I20260924 10:12:39.368117 25992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.360887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:57779 (local address 127.6.2.193:46407)
0924 10:12:39.361099 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:39.361105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:39.361130 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:12:39.361476 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:39.361482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:39.361577 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:39.361684 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:39.362175 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.363173 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.365223 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.366316 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.366420 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.366497 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:39.367698 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0924 10:12:39.367846 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:12:39.368683 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.360821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:33029 (local address 127.6.2.195:52769)
0924 10:12:39.361718 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:39.361738 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:39.361890 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:39.362274 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:39.362280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:39.362306 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:39.362682 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:39.362693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.364265 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.364268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:39.365887 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:39.365901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.367793 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.367800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:39.367979 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.368014 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:12:39.368121 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 10:12:39.368270 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"spinlock_wait_cycles":1920,"thread_start_us":503,"threads_started":1}
I20260924 10:12:39.368934 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.360783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:52769 (local address 127.6.2.194:33029)
0924 10:12:39.361114 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:39.361122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:39.361834 (+ 712us) server_negotiation.cc:407] Connection header received
0924 10:12:39.362031 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:39.362036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:39.362132 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:39.362229 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:39.362891 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.364090 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.366059 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.367323 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.367466 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.367587 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:39.368179 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 10:12:39.368661 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260924 10:12:39.369165 25661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4909efb514432dab83e02fbf990ec1" candidate_uuid: "bf9283a7092942508761244446c78155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4deea200ad84fc48076b6dfe4682082" is_pre_election: true
I20260924 10:12:39.369736 25661 raft_consensus.cc:2454] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9283a7092942508761244446c78155 in term 0.
I20260924 10:12:39.369786 25795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4909efb514432dab83e02fbf990ec1" candidate_uuid: "bf9283a7092942508761244446c78155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b43cf9d7a6451488d805a0f4894482" is_pre_election: true
I20260924 10:12:39.370301 25795 raft_consensus.cc:2454] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9283a7092942508761244446c78155 in term 0.
I20260924 10:12:39.370433 25865 leader_election.cc:302] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf9283a7092942508761244446c78155, e4deea200ad84fc48076b6dfe4682082; no voters:
I20260924 10:12:39.370916 26000 raft_consensus.cc:2781] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:12:39.371097 26000 raft_consensus.cc:482] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:12:39.371227 26000 raft_consensus.cc:3037] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:39.373507 26000 raft_consensus.cc:504] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b43cf9d7a6451488d805a0f4894482" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 33029 } } peers { permanent_uuid: "bf9283a7092942508761244446c78155" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 32855 } } peers { permanent_uuid: "e4deea200ad84fc48076b6dfe4682082" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 46407 } }
I20260924 10:12:39.374123 26000 leader_election.cc:288] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [CANDIDATE]: Term 1 election: Requested vote from peers e3b43cf9d7a6451488d805a0f4894482 (127.6.2.194:33029), e4deea200ad84fc48076b6dfe4682082 (127.6.2.193:46407)
I20260924 10:12:39.374671 25795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4909efb514432dab83e02fbf990ec1" candidate_uuid: "bf9283a7092942508761244446c78155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b43cf9d7a6451488d805a0f4894482"
I20260924 10:12:39.374670 25661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4909efb514432dab83e02fbf990ec1" candidate_uuid: "bf9283a7092942508761244446c78155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4deea200ad84fc48076b6dfe4682082"
I20260924 10:12:39.374877 25795 raft_consensus.cc:3037] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:39.374900 25661 raft_consensus.cc:3037] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:39.378084 25795 raft_consensus.cc:2454] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9283a7092942508761244446c78155 in term 1.
I20260924 10:12:39.378084 25661 raft_consensus.cc:2454] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9283a7092942508761244446c78155 in term 1.
I20260924 10:12:39.378504 25865 leader_election.cc:302] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf9283a7092942508761244446c78155, e4deea200ad84fc48076b6dfe4682082; no voters:
I20260924 10:12:39.378795 26000 raft_consensus.cc:2781] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:39.379287 26000 raft_consensus.cc:686] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [term 1 LEADER]: Becoming Leader. State: Replica: bf9283a7092942508761244446c78155, State: Running, Role: LEADER
I20260924 10:12:39.379980 26000 consensus_queue.cc:237] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b43cf9d7a6451488d805a0f4894482" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 33029 } } peers { permanent_uuid: "bf9283a7092942508761244446c78155" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 32855 } } peers { permanent_uuid: "e4deea200ad84fc48076b6dfe4682082" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 46407 } }
I20260924 10:12:39.384756 25516 catalog_manager.cc:5495] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9283a7092942508761244446c78155 (127.6.2.195). New cstate: current_term: 1 leader_uuid: "bf9283a7092942508761244446c78155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b43cf9d7a6451488d805a0f4894482" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 33029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9283a7092942508761244446c78155" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 32855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4deea200ad84fc48076b6dfe4682082" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260924 10:12:39.417316 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.412133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.6.2.195:32855)
0924 10:12:39.412349 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:39.412353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:39.412372 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:12:39.412543 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:39.412547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:39.412621 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:39.412742 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:12:39.413117 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.413802 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.415230 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.415610 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.415703 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.416186 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:12:39.416680 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:12:39.416796 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:39.416952 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 10:12:39.417017 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:12:39.426064 25661 raft_consensus.cc:1261] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082 [term 1 FOLLOWER]: Refusing update from remote peer bf9283a7092942508761244446c78155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:12:39.426067 25795 raft_consensus.cc:1261] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482 [term 1 FOLLOWER]: Refusing update from remote peer bf9283a7092942508761244446c78155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:12:39.427228 26000 consensus_queue.cc:1035] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b43cf9d7a6451488d805a0f4894482" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:12:39.427595 26003 consensus_queue.cc:1035] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4deea200ad84fc48076b6dfe4682082" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:12:39.439471 26006 mvcc.cc:205] Tried to move back new op lower bound from 7332842534598090752 to 7332842534424170496. Current Snapshot: MvccSnapshot[applied={T|T < 7332842534598090752}]
I20260924 10:12:39.458199 25517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:12:39.458469 25517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:12:39.462159 25517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 495e791597ec48cab0283552b3ebeca9: Sending DeleteTablet for 3 replicas of tablet 1c4909efb514432dab83e02fbf990ec1
I20260924 10:12:39.463279 25909 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4909efb514432dab83e02fbf990ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:39 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:41896
I20260924 10:12:39.463280 25775 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4909efb514432dab83e02fbf990ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:39 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:44092
I20260924 10:12:39.464898 25641 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4909efb514432dab83e02fbf990ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:39 UTC) from {username='kudu', principal='kudu/127.6.2.254@KRBTEST.COM'} at 127.0.0.1:59604
I20260924 10:12:39.469349 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 25578
I20260924 10:12:39.476781 26017 tablet_replica.cc:330] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482: stopping tablet replica
I20260924 10:12:39.477473 26017 raft_consensus.cc:2229] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:12:39.478101 26017 raft_consensus.cc:2258] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:39.478960 26016 tablet_replica.cc:330] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155: stopping tablet replica
I20260924 10:12:39.479417 26016 raft_consensus.cc:2229] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:12:39.480129 26016 raft_consensus.cc:2258] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:39.481283 26017 ts_tablet_manager.cc:1905] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:39.482263 26016 ts_tablet_manager.cc:1905] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:39.489924 26016 ts_tablet_manager.cc:1918] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:39.490108 26016 log.cc:1196] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/1c4909efb514432dab83e02fbf990ec1
I20260924 10:12:39.490680 26016 ts_tablet_manager.cc:1939] T 1c4909efb514432dab83e02fbf990ec1 P bf9283a7092942508761244446c78155: Deleting consensus metadata
I20260924 10:12:39.491389 26017 ts_tablet_manager.cc:1918] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:39.491673 26017 log.cc:1196] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/1c4909efb514432dab83e02fbf990ec1
I20260924 10:12:39.492337 26017 ts_tablet_manager.cc:1939] T 1c4909efb514432dab83e02fbf990ec1 P e3b43cf9d7a6451488d805a0f4894482: Deleting consensus metadata
I20260924 10:12:39.492841 25503 catalog_manager.cc:4841] TS bf9283a7092942508761244446c78155 (127.6.2.195:32855): tablet 1c4909efb514432dab83e02fbf990ec1 (table test-table [id=d352691869d64115947a4a3187bc1e24]) successfully deleted
I20260924 10:12:39.494405 26018 tablet_replica.cc:330] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082: stopping tablet replica
I20260924 10:12:39.494449 25502 catalog_manager.cc:4841] TS e3b43cf9d7a6451488d805a0f4894482 (127.6.2.194:33029): tablet 1c4909efb514432dab83e02fbf990ec1 (table test-table [id=d352691869d64115947a4a3187bc1e24]) successfully deleted
I20260924 10:12:39.494899 26018 raft_consensus.cc:2229] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:12:39.495491 26018 raft_consensus.cc:2258] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:39.497642 26018 ts_tablet_manager.cc:1905] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 10:12:39 dist-test-slave-s3n3 krb5kdc[25472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.193@KRBTEST.COM
I20260924 10:12:39.504769 26018 ts_tablet_manager.cc:1918] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:39.504954 26018 log.cc:1196] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790244437048964-6155-0/minicluster-data/ts-0/wal/wals/1c4909efb514432dab83e02fbf990ec1
I20260924 10:12:39.505548 26018 ts_tablet_manager.cc:1939] T 1c4909efb514432dab83e02fbf990ec1 P e4deea200ad84fc48076b6dfe4682082: Deleting consensus metadata
I20260924 10:12:39.507486 25504 catalog_manager.cc:4841] TS e4deea200ad84fc48076b6dfe4682082 (127.6.2.193:46407): tablet 1c4909efb514432dab83e02fbf990ec1 (table test-table [id=d352691869d64115947a4a3187bc1e24]) successfully deleted
I20260924 10:12:39.511263 25992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.470745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.6.2.193:46407)
0924 10:12:39.470985 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:39.470995 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:39.493698 (+ 22703us) server_negotiation.cc:407] Connection header received
0924 10:12:39.493903 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:39.493911 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:39.494054 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:39.494174 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:39.497034 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.497876 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.500042 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.500475 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.500647 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.504906 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:39.504922 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:39.504940 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:39.505024 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:39.508436 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:39.509134 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:39.509143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:39.509147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:39.509287 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:39.509678 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:39.509683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:39.509686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:39.510031 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:39.510340 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:39.510638 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 10:12:39.510855 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:12:39.511983 25701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:39.614714 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 25578
I20260924 10:12:39.633126 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 25712
Sep 24 10:12:39 dist-test-slave-s3n3 krb5kdc[25472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.194@KRBTEST.COM
I20260924 10:12:39.650563 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.633945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.6.2.194:33029)
0924 10:12:39.634147 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:39.634154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:39.634176 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:12:39.634391 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:39.634397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:39.634483 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:39.634600 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:39.636056 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.637217 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.638491 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.639217 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.639352 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.643749 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:39.643769 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:39.643792 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:39.643855 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:39.647614 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:39.648292 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:39.648302 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:39.648305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:39.648452 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:39.648811 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:39.648818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:39.648823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:39.649318 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:39.649614 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:39.649946 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 10:12:39.650175 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:12:39.651400 25835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:39.741997 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 25712
I20260924 10:12:39.765736 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 25846
Sep 24 10:12:39 dist-test-slave-s3n3 krb5kdc[25472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.195@KRBTEST.COM
I20260924 10:12:39.782615 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:39.766599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.6.2.195:32855)
0924 10:12:39.766816 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:39.766823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:39.766843 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:12:39.767034 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:39.767043 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:39.767153 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:39.767274 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:39.768504 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.769345 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.770718 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:39.771379 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:39.771525 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:39.776257 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:39.776280 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:39.776301 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:39.776348 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:39.779712 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:39.780461 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:39.780470 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:39.780472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:39.780600 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:39.780996 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:39.781002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:39.781006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:39.781372 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:39.781694 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:39.782010 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 10:12:39.782253 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 10:12:39.783448 25969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:39.909677 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 25846
I20260924 10:12:39.929563 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 25485
I20260924 10:12:39.930639 25547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:40.043255 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 25485
2026-09-24T10:12:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4415 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T10:12:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:40Z Disabled control of system clock
I20260924 10:12:40.122609 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:32861
--webserver_interface=127.6.2.254
--webserver_port=0
--builtin_ntp_servers=127.6.2.212:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:32861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:12:40.451588 26027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:40.451871 26027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:40.451954 26027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:40.459254 26027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:40.459360 26027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:40.459398 26027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:40.459447 26027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:40.459484 26027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:40.468869 26027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:32861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:32861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:40.470114 26027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:40.471907 26027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:40.483476 26033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:40.483589 26032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:40.483528 26035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:40.484429 26027 server_base.cc:1034] running on GCE node
I20260924 10:12:40.485167 26027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:40.486631 26027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:40.487836 26027 hybrid_clock.cc:648] HybridClock initialized: now 1790244760487806 us; error 33 us; skew 500 ppm
I20260924 10:12:40.488190 26027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:12:40.490530 26027 webserver.cc:460] Webserver started at http://127.6.2.254:33107/ using document root <none> and password file <none>
I20260924 10:12:40.491267 26027 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:40.491358 26027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:40.491662 26027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:40.494586 26027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "011788c98de74b7b8b099776a0e45f24"
format_stamp: "Formatted at 2026-09-24 10:12:40 on dist-test-slave-s3n3"
server_key: "c8002231c0cddef8793ddaf09559f243"
server_key_iv: "a997655a287bfab28e648a2c607ec449"
server_key_version: "encryptionkey@0"
I20260924 10:12:40.495841 26027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "011788c98de74b7b8b099776a0e45f24"
format_stamp: "Formatted at 2026-09-24 10:12:40 on dist-test-slave-s3n3"
server_key: "c8002231c0cddef8793ddaf09559f243"
server_key_iv: "a997655a287bfab28e648a2c607ec449"
server_key_version: "encryptionkey@0"
I20260924 10:12:40.501852 26027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260924 10:12:40.505290 26041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:40.506616 26027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:12:40.506757 26027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "011788c98de74b7b8b099776a0e45f24"
format_stamp: "Formatted at 2026-09-24 10:12:40 on dist-test-slave-s3n3"
server_key: "c8002231c0cddef8793ddaf09559f243"
server_key_iv: "a997655a287bfab28e648a2c607ec449"
server_key_version: "encryptionkey@0"
I20260924 10:12:40.506946 26027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:40.533252 26027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:40.534113 26027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:40.534359 26027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:40.553027 26027 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:32861
I20260924 10:12:40.553052 26092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:32861 every 8 connection(s)
I20260924 10:12:40.554368 26027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:40.556826 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 26027
I20260924 10:12:40.557008 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:40.557448 6155 external_mini_cluster.cc:1428] Setting key e22a081beae7f4d25317f0dabf73d869
I20260924 10:12:40.558511 26093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:40.570978 26096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:40.559284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.6.2.254:32861)
0924 10:12:40.560403 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:40.560421 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:40.560480 (+ 59us) server_negotiation.cc:407] Connection header received
0924 10:12:40.562217 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:40.562237 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:40.562717 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:40.563130 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:40.563694 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:40.565006 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:40.566417 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:40.567135 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:40.567328 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:40.568594 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:40.568601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:40.568618 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:40.569238 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:40.569326 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:40.569349 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:40.569487 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:40.569706 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 10:12:40.570035 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":391,"threads_started":1}
I20260924 10:12:40.571583 26093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24: Bootstrap starting.
I20260924 10:12:40.574867 26093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:40.575987 26093 log.cc:824] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:40.578742 26093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24: No bootstrap required, opened a new log
I20260924 10:12:40.586493 26093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011788c98de74b7b8b099776a0e45f24" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 32861 } }
I20260924 10:12:40.586987 26093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:40.587114 26093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011788c98de74b7b8b099776a0e45f24, State: Initialized, Role: FOLLOWER
I20260924 10:12:40.587705 26093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011788c98de74b7b8b099776a0e45f24" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 32861 } }
I20260924 10:12:40.587949 26093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:40.588100 26093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:40.588282 26093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:40.590523 26093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011788c98de74b7b8b099776a0e45f24" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 32861 } }
I20260924 10:12:40.591122 26093 leader_election.cc:302] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 011788c98de74b7b8b099776a0e45f24; no voters:
I20260924 10:12:40.591866 26093 leader_election.cc:288] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:40.592003 26098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:40.592708 26098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [term 1 LEADER]: Becoming Leader. State: Replica: 011788c98de74b7b8b099776a0e45f24, State: Running, Role: LEADER
I20260924 10:12:40.593257 26098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011788c98de74b7b8b099776a0e45f24" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 32861 } }
I20260924 10:12:40.594192 26093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:40.596138 26099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "011788c98de74b7b8b099776a0e45f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011788c98de74b7b8b099776a0e45f24" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 32861 } } }
I20260924 10:12:40.596246 26100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 011788c98de74b7b8b099776a0e45f24. Latest consensus state: current_term: 1 leader_uuid: "011788c98de74b7b8b099776a0e45f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011788c98de74b7b8b099776a0e45f24" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 32861 } } }
I20260924 10:12:40.596683 26100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:40.596691 26099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:40.598495 26103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:40.606473 26103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:40.614774 26103 catalog_manager.cc:1347] Generated new cluster ID: 090e780b0eac40d9aede1026816c44e0
I20260924 10:12:40.614892 26103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:40.641712 26103 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:40.643267 26103 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:40.652266 26103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24: Generated new TSK 0
I20260924 10:12:40.653107 26103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:12:40.659408 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.193:0
--local_ip_for_outbound_sockets=127.6.2.193
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:32861
--builtin_ntp_servers=127.6.2.212:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:12:40.991215 26117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:40.991549 26117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:40.991705 26117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:40.998970 26117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:40.999127 26117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:40.999301 26117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.193
I20260924 10:12:41.008790 26117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.2.193
--webserver_port=0
--tserver_master_addrs=127.6.2.254:32861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.193
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:41.010217 26117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:41.012116 26117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:41.025501 26122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:41.025558 26123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:41.025630 26125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:41.027233 26117 server_base.cc:1034] running on GCE node
I20260924 10:12:41.027874 26117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:41.028882 26117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:41.030094 26117 hybrid_clock.cc:648] HybridClock initialized: now 1790244761030032 us; error 51 us; skew 500 ppm
I20260924 10:12:41.030413 26117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:12:41.032670 26117 webserver.cc:460] Webserver started at http://127.6.2.193:38687/ using document root <none> and password file <none>
I20260924 10:12:41.033342 26117 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:41.033465 26117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:41.033779 26117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:41.036688 26117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/data/instance:
uuid: "760e23cd453046f9877517be8a8f53fa"
format_stamp: "Formatted at 2026-09-24 10:12:41 on dist-test-slave-s3n3"
server_key: "40c5a63fee4f4473fb0f6dd356d56b5b"
server_key_iv: "a6fc73a94c748341be9baba37ec32da4"
server_key_version: "encryptionkey@0"
I20260924 10:12:41.037477 26117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance:
uuid: "760e23cd453046f9877517be8a8f53fa"
format_stamp: "Formatted at 2026-09-24 10:12:41 on dist-test-slave-s3n3"
server_key: "40c5a63fee4f4473fb0f6dd356d56b5b"
server_key_iv: "a6fc73a94c748341be9baba37ec32da4"
server_key_version: "encryptionkey@0"
I20260924 10:12:41.042605 26117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 10:12:41.046005 26131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:41.047384 26117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:12:41.047562 26117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/wal
uuid: "760e23cd453046f9877517be8a8f53fa"
format_stamp: "Formatted at 2026-09-24 10:12:41 on dist-test-slave-s3n3"
server_key: "40c5a63fee4f4473fb0f6dd356d56b5b"
server_key_iv: "a6fc73a94c748341be9baba37ec32da4"
server_key_version: "encryptionkey@0"
I20260924 10:12:41.047776 26117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:41.059672 26117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:41.060490 26117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:41.060766 26117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:41.061691 26117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:41.063218 26117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:41.063334 26117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:41.063475 26117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:41.063565 26117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:41.097142 26117 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.193:42973
I20260924 10:12:41.097230 26243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.193:42973 every 8 connection(s)
I20260924 10:12:41.098630 26117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/data/info.pb
I20260924 10:12:41.107853 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 26117
I20260924 10:12:41.108033 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-0/wal/instance
I20260924 10:12:41.108498 6155 external_mini_cluster.cc:1428] Setting key 6aef8c15c4656e59d12547f97cff4171
I20260924 10:12:41.111856 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.194:0
--local_ip_for_outbound_sockets=127.6.2.194
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:32861
--builtin_ntp_servers=127.6.2.212:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:12:41.112437 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:41.101415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:51397 (local address 127.6.2.254:32861)
0924 10:12:41.101927 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:41.101933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:41.103169 (+ 1236us) server_negotiation.cc:407] Connection header received
0924 10:12:41.105043 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:41.105047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:41.105123 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:41.105223 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:41.106342 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:41.107203 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:41.109045 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:41.109662 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:41.109867 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:41.111490 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:41.111497 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:41.111501 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:41.111522 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:41.111577 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:41.111595 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:41.111705 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:41.112005 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 10:12:41.112103 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":259,"threads_started":1}
I20260924 10:12:41.113607 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:41.102024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:32861 (local address 127.6.2.193:51397)
0924 10:12:41.102967 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:41.103009 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:41.104784 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:41.105512 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:41.105521 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:41.105612 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:41.106169 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:41.106185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:41.107346 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:41.107351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:41.108877 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:41.108892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:41.109915 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:41.109926 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:41.110211 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:41.111081 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:12:41.111086 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:41.111143 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:12:41.111257 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:41.111764 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:41.111776 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:41.111786 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:12:41.111991 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 10:12:41.112396 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":311,"threads_started":1}
I20260924 10:12:41.115499 26244 heartbeater.cc:347] Connected to a master server at 127.6.2.254:32861
I20260924 10:12:41.115840 26244 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:41.116772 26244 heartbeater.cc:511] Master 127.6.2.254:32861 requested a full tablet report, sending...
I20260924 10:12:41.118983 26058 ts_manager.cc:194] Registered new tserver with Master: 760e23cd453046f9877517be8a8f53fa (127.6.2.193:42973)
I20260924 10:12:41.120585 26058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.2.193:51397
W20260924 10:12:41.447520 26248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:41.447841 26248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:41.447938 26248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:41.455817 26248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:41.455921 26248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:41.456032 26248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.194
I20260924 10:12:41.465162 26248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.2.194
--webserver_port=0
--tserver_master_addrs=127.6.2.254:32861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.194
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:41.466411 26248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:41.468291 26248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:41.481240 26256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:41.481236 26253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:41.481484 26254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:41.481969 26248 server_base.cc:1034] running on GCE node
I20260924 10:12:41.482512 26248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:41.483479 26248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:41.484678 26248 hybrid_clock.cc:648] HybridClock initialized: now 1790244761484638 us; error 36 us; skew 500 ppm
I20260924 10:12:41.484963 26248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:12:41.487618 26248 webserver.cc:460] Webserver started at http://127.6.2.194:36139/ using document root <none> and password file <none>
I20260924 10:12:41.488255 26248 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:41.488341 26248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:41.488651 26248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:41.491483 26248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/data/instance:
uuid: "a9e7398c7ed447bbb3edb12a92450398"
format_stamp: "Formatted at 2026-09-24 10:12:41 on dist-test-slave-s3n3"
server_key: "6c1162155783a9a8c2d60f7c70e97bc1"
server_key_iv: "a34a4dc917677fa4609eea4cfa35c367"
server_key_version: "encryptionkey@0"
I20260924 10:12:41.492266 26248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance:
uuid: "a9e7398c7ed447bbb3edb12a92450398"
format_stamp: "Formatted at 2026-09-24 10:12:41 on dist-test-slave-s3n3"
server_key: "6c1162155783a9a8c2d60f7c70e97bc1"
server_key_iv: "a34a4dc917677fa4609eea4cfa35c367"
server_key_version: "encryptionkey@0"
I20260924 10:12:41.497197 26248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 10:12:41.500854 26262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:41.502264 26248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:12:41.502393 26248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/wal
uuid: "a9e7398c7ed447bbb3edb12a92450398"
format_stamp: "Formatted at 2026-09-24 10:12:41 on dist-test-slave-s3n3"
server_key: "6c1162155783a9a8c2d60f7c70e97bc1"
server_key_iv: "a34a4dc917677fa4609eea4cfa35c367"
server_key_version: "encryptionkey@0"
I20260924 10:12:41.502581 26248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:41.520915 26248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:41.521729 26248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:41.521956 26248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:41.522877 26248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:41.524438 26248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:41.524538 26248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:41.524618 26248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:41.524672 26248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:41.559618 26248 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.194:40969
I20260924 10:12:41.559674 26374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.194:40969 every 8 connection(s)
I20260924 10:12:41.561264 26248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/data/info.pb
I20260924 10:12:41.570003 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 26248
I20260924 10:12:41.570194 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/wal/instance
I20260924 10:12:41.570636 6155 external_mini_cluster.cc:1428] Setting key 463b483f7da98382e8fc25565ac351eb
I20260924 10:12:41.573627 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.2.195:0
--local_ip_for_outbound_sockets=127.6.2.195
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:32861
--builtin_ntp_servers=127.6.2.212:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:12:41.575101 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:41.564012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.194:49877 (local address 127.6.2.254:32861)
0924 10:12:41.564189 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:41.564194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:41.565765 (+ 1571us) server_negotiation.cc:407] Connection header received
0924 10:12:41.567661 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:41.567666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:41.567760 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:41.567878 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:41.568909 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:41.569710 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:41.572145 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:41.572722 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:41.572866 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:41.574292 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:41.574296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:41.574300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:41.574317 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:41.574355 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:41.574368 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:41.574464 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:41.574687 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 10:12:41.574787 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:12:41.576174 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:41.564651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:32861 (local address 127.6.2.194:49877)
0924 10:12:41.565600 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:41.565640 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:41.567401 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:41.568104 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:41.568112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:41.568193 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:41.568748 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:41.568763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:41.569890 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:41.569896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:41.571981 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:41.571997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:41.572958 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:41.572966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:41.573191 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:41.573964 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:12:41.573966 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:41.574006 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:12:41.574084 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:41.574461 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:41.574474 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:41.574484 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:12:41.574676 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 10:12:41.575122 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":341,"threads_started":1}
I20260924 10:12:41.577526 26375 heartbeater.cc:347] Connected to a master server at 127.6.2.254:32861
I20260924 10:12:41.577854 26375 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:41.578794 26375 heartbeater.cc:511] Master 127.6.2.254:32861 requested a full tablet report, sending...
I20260924 10:12:41.580410 26058 ts_manager.cc:194] Registered new tserver with Master: a9e7398c7ed447bbb3edb12a92450398 (127.6.2.194:40969)
I20260924 10:12:41.581657 26058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.2.194:49877
W20260924 10:12:41.905085 26378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:41.905388 26378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:41.905530 26378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:41.912875 26378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:41.912981 26378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:41.913143 26378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.2.195
I20260924 10:12:41.922401 26378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.2.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.2.195
--webserver_port=0
--tserver_master_addrs=127.6.2.254:32861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.2.195
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:41.923803 26378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:41.925596 26378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:41.939630 26383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:41.939642 26386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:41.939742 26384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:41.940281 26378 server_base.cc:1034] running on GCE node
I20260924 10:12:41.941061 26378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:41.942185 26378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:41.943440 26378 hybrid_clock.cc:648] HybridClock initialized: now 1790244761943405 us; error 49 us; skew 500 ppm
I20260924 10:12:41.943843 26378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 10:12:41.946182 26378 webserver.cc:460] Webserver started at http://127.6.2.195:42805/ using document root <none> and password file <none>
I20260924 10:12:41.946892 26378 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:41.947027 26378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:41.947376 26378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:41.950277 26378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/data/instance:
uuid: "d3124c0d368642ea9667fda9312607c3"
format_stamp: "Formatted at 2026-09-24 10:12:41 on dist-test-slave-s3n3"
server_key: "f3191a6c33b83de5d487c2f5af5c2dc5"
server_key_iv: "42d0ab4e1a5c55add0edb2c586674cd6"
server_key_version: "encryptionkey@0"
I20260924 10:12:41.951225 26378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance:
uuid: "d3124c0d368642ea9667fda9312607c3"
format_stamp: "Formatted at 2026-09-24 10:12:41 on dist-test-slave-s3n3"
server_key: "f3191a6c33b83de5d487c2f5af5c2dc5"
server_key_iv: "42d0ab4e1a5c55add0edb2c586674cd6"
server_key_version: "encryptionkey@0"
I20260924 10:12:41.956393 26378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 10:12:41.959990 26392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:41.961303 26378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:12:41.961477 26378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/wal
uuid: "d3124c0d368642ea9667fda9312607c3"
format_stamp: "Formatted at 2026-09-24 10:12:41 on dist-test-slave-s3n3"
server_key: "f3191a6c33b83de5d487c2f5af5c2dc5"
server_key_iv: "42d0ab4e1a5c55add0edb2c586674cd6"
server_key_version: "encryptionkey@0"
I20260924 10:12:41.961700 26378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:42.004519 26378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:42.005391 26378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:42.005723 26378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:42.006917 26378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:12:42.008519 26378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:12:42.008643 26378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:42.008780 26378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:12:42.008878 26378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:42.042219 26378 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.195:33661
I20260924 10:12:42.042271 26504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.195:33661 every 8 connection(s)
I20260924 10:12:42.043920 26378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/data/info.pb
I20260924 10:12:42.050741 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 26378
I20260924 10:12:42.050920 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/wal/instance
I20260924 10:12:42.051535 6155 external_mini_cluster.cc:1428] Setting key d9333046199217cffeade8df857607ef
I20260924 10:12:42.056835 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.046485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.195:52051 (local address 127.6.2.254:32861)
0924 10:12:42.046669 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:42.046674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:42.048098 (+ 1424us) server_negotiation.cc:407] Connection header received
0924 10:12:42.049873 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:42.049878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:42.049970 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:42.050082 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:42.051504 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.052405 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.054405 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.054813 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.054918 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.056185 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:42.056190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:42.056193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:42.056210 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:42.056241 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:42.056252 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:42.056333 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:42.056522 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 10:12:42.056601 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:12:42.057824 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.046904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:32861 (local address 127.6.2.195:52051)
0924 10:12:42.047951 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:42.047985 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:42.049601 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:42.050299 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:42.050331 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:42.050451 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:12:42.051305 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:42.051330 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.052591 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.052597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:42.054265 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:42.054279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.055000 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.055005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:42.055287 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.055905 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:12:42.055908 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:12:42.055944 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:12:42.056019 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:12:42.056337 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:12:42.056345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:12:42.056350 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:12:42.056487 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 10:12:42.056791 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":433,"threads_started":1}
I20260924 10:12:42.059114 26505 heartbeater.cc:347] Connected to a master server at 127.6.2.254:32861
I20260924 10:12:42.059437 26505 heartbeater.cc:464] Registering TS with master...
I20260924 10:12:42.060156 26505 heartbeater.cc:511] Master 127.6.2.254:32861 requested a full tablet report, sending...
I20260924 10:12:42.061568 26058 ts_manager.cc:194] Registered new tserver with Master: d3124c0d368642ea9667fda9312607c3 (127.6.2.195:33661)
I20260924 10:12:42.062350 26058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.2.195:52051
I20260924 10:12:42.066687 6155 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:12:42.077170 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.070964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.6.2.254:32861)
0924 10:12:42.071236 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:42.071242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:42.071537 (+ 295us) server_negotiation.cc:407] Connection header received
0924 10:12:42.071628 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:42.071633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:42.071718 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:42.071829 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:42.072278 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.073169 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.074631 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.075018 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.075214 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.076331 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:42.076336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:42.076340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:42.076358 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:42.076391 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:42.076404 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:42.076483 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:42.076631 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 10:12:42.076728 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 10:12:42.081238 26058 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:12:42.088704 26058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:12:42.116313 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.101695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42973 (local address 127.0.0.1:48336)
0924 10:12:42.104420 (+ 2725us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:42.104447 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:42.104628 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:42.105907 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:42.105918 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:42.105952 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:42.106453 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:42.106472 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.109218 (+ 2746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.109225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:42.112943 (+ 3718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:42.112960 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.114932 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.114941 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:42.115256 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.115370 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 10:12:42.115554 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 10:12:42.115907 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2526,"mutex_wait_us":774,"thread_start_us":828,"threads_started":1}
I20260924 10:12:42.116498 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.101523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.6.2.194:40969)
0924 10:12:42.107080 (+ 5557us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:42.107089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:42.107117 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:12:42.107222 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:42.107230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:42.107464 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:42.108361 (+ 897us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:42.108578 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.110136 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.113198 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.114861 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.114973 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.115293 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:42.115941 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 10:12:42.116135 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5386,"thread_start_us":5263,"threads_started":1}
I20260924 10:12:42.116917 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.102292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.6.2.193:42973)
0924 10:12:42.103618 (+ 1326us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:42.103630 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:42.104805 (+ 1175us) server_negotiation.cc:407] Connection header received
0924 10:12:42.104962 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:42.104975 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:42.105291 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:42.105533 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:42.107135 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.109022 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.113200 (+ 4178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.114668 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.114852 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.116262 (+ 1410us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:42.116343 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 10:12:42.116519 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":936,"threads_started":1}
I20260924 10:12:42.117334 26519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.101311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:40969 (local address 127.0.0.1:54054)
0924 10:12:42.102354 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:42.102403 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:42.102586 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:42.107694 (+ 5108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:42.107700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:42.107728 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:42.108163 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:42.108177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.110311 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.110316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:42.112943 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:42.112960 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.115491 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.115498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:42.115701 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.115733 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:12:42.116763 (+ 1030us) client_negotiation.cc:240] Negotiation successful
0924 10:12:42.116933 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":63,"thread_start_us":2719,"threads_started":1}
I20260924 10:12:42.121753 26179 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0ad63897b242f7b00478e283017b3d (DEFAULT_TABLE table=test-table [id=fe49ac56cd544217a040f1e57ee556ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:42.121706 26310 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0ad63897b242f7b00478e283017b3d (DEFAULT_TABLE table=test-table [id=fe49ac56cd544217a040f1e57ee556ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:42.123193 26310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0ad63897b242f7b00478e283017b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:42.123234 26179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0ad63897b242f7b00478e283017b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:42.123762 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.104372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:33661 (local address 127.0.0.1:46152)
0924 10:12:42.111974 (+ 7602us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:42.111994 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:42.112109 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:42.114377 (+ 2268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:42.114383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:42.114410 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:42.114784 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:42.114798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.119010 (+ 4212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.119015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:42.121013 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:42.121029 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.122901 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.122908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:42.123139 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.123185 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 10:12:42.123295 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 10:12:42.123439 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7464,"thread_start_us":7311,"threads_started":1}
I20260924 10:12:42.123927 26244 heartbeater.cc:503] Master 127.6.2.254:32861 was elected leader, sending a full tablet report...
I20260924 10:12:42.124652 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.104748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.6.2.195:33661)
0924 10:12:42.113710 (+ 8962us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:42.113719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:42.113758 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:12:42.113859 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:42.113867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:42.114099 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:42.115167 (+ 1068us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:42.115488 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.118849 (+ 3361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.121260 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.122664 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.122825 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.123106 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:42.124082 (+ 976us) server_negotiation.cc:299] Negotiation successful
0924 10:12:42.124262 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8776,"thread_start_us":8603,"threads_started":1}
I20260924 10:12:42.134446 26440 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0ad63897b242f7b00478e283017b3d (DEFAULT_TABLE table=test-table [id=fe49ac56cd544217a040f1e57ee556ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:12:42.135757 26523 tablet_bootstrap.cc:492] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa: Bootstrap starting.
I20260924 10:12:42.136833 26440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0ad63897b242f7b00478e283017b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:42.137915 26524 tablet_bootstrap.cc:492] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398: Bootstrap starting.
I20260924 10:12:42.139875 26523 tablet_bootstrap.cc:654] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:42.141247 26523 log.cc:824] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:42.141855 26524 tablet_bootstrap.cc:654] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:42.143270 26524 log.cc:824] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:42.145849 26523 tablet_bootstrap.cc:492] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa: No bootstrap required, opened a new log
I20260924 10:12:42.145961 26524 tablet_bootstrap.cc:492] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398: No bootstrap required, opened a new log
I20260924 10:12:42.146138 26523 ts_tablet_manager.cc:1397] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 10:12:42.146241 26524 ts_tablet_manager.cc:1397] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260924 10:12:42.152014 26527 tablet_bootstrap.cc:492] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3: Bootstrap starting.
I20260924 10:12:42.155593 26523 raft_consensus.cc:348] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e7398c7ed447bbb3edb12a92450398" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40969 } } peers { permanent_uuid: "760e23cd453046f9877517be8a8f53fa" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42973 } } peers { permanent_uuid: "d3124c0d368642ea9667fda9312607c3" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 33661 } }
I20260924 10:12:42.155989 26523 raft_consensus.cc:374] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:42.156078 26523 raft_consensus.cc:729] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 760e23cd453046f9877517be8a8f53fa, State: Initialized, Role: FOLLOWER
I20260924 10:12:42.156174 26527 tablet_bootstrap.cc:654] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:42.156311 26524 raft_consensus.cc:348] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e7398c7ed447bbb3edb12a92450398" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40969 } } peers { permanent_uuid: "760e23cd453046f9877517be8a8f53fa" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42973 } } peers { permanent_uuid: "d3124c0d368642ea9667fda9312607c3" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 33661 } }
I20260924 10:12:42.156821 26524 raft_consensus.cc:374] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:42.156716 26523 consensus_queue.cc:260] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e7398c7ed447bbb3edb12a92450398" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40969 } } peers { permanent_uuid: "760e23cd453046f9877517be8a8f53fa" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42973 } } peers { permanent_uuid: "d3124c0d368642ea9667fda9312607c3" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 33661 } }
I20260924 10:12:42.156930 26524 raft_consensus.cc:729] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e7398c7ed447bbb3edb12a92450398, State: Initialized, Role: FOLLOWER
I20260924 10:12:42.157615 26527 log.cc:824] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:42.157682 26524 consensus_queue.cc:260] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e7398c7ed447bbb3edb12a92450398" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40969 } } peers { permanent_uuid: "760e23cd453046f9877517be8a8f53fa" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42973 } } peers { permanent_uuid: "d3124c0d368642ea9667fda9312607c3" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 33661 } }
I20260924 10:12:42.158582 26523 ts_tablet_manager.cc:1428] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 10:12:42.159489 26375 heartbeater.cc:503] Master 127.6.2.254:32861 was elected leader, sending a full tablet report...
I20260924 10:12:42.160738 26524 ts_tablet_manager.cc:1428] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260924 10:12:42.163362 26527 tablet_bootstrap.cc:492] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3: No bootstrap required, opened a new log
I20260924 10:12:42.163668 26527 ts_tablet_manager.cc:1397] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260924 10:12:42.175228 26527 raft_consensus.cc:348] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e7398c7ed447bbb3edb12a92450398" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40969 } } peers { permanent_uuid: "760e23cd453046f9877517be8a8f53fa" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42973 } } peers { permanent_uuid: "d3124c0d368642ea9667fda9312607c3" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 33661 } }
I20260924 10:12:42.175801 26527 raft_consensus.cc:374] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:42.175935 26527 raft_consensus.cc:729] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3124c0d368642ea9667fda9312607c3, State: Initialized, Role: FOLLOWER
I20260924 10:12:42.176741 26527 consensus_queue.cc:260] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e7398c7ed447bbb3edb12a92450398" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40969 } } peers { permanent_uuid: "760e23cd453046f9877517be8a8f53fa" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42973 } } peers { permanent_uuid: "d3124c0d368642ea9667fda9312607c3" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 33661 } }
I20260924 10:12:42.178143 26505 heartbeater.cc:503] Master 127.6.2.254:32861 was elected leader, sending a full tablet report...
I20260924 10:12:42.178754 26527 ts_tablet_manager.cc:1428] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260924 10:12:42.194083 26528 raft_consensus.cc:482] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:12:42.194366 26528 raft_consensus.cc:504] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e7398c7ed447bbb3edb12a92450398" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40969 } } peers { permanent_uuid: "760e23cd453046f9877517be8a8f53fa" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42973 } } peers { permanent_uuid: "d3124c0d368642ea9667fda9312607c3" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 33661 } }
I20260924 10:12:42.196151 26528 leader_election.cc:288] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9e7398c7ed447bbb3edb12a92450398 (127.6.2.194:40969), d3124c0d368642ea9667fda9312607c3 (127.6.2.195:33661)
I20260924 10:12:42.203979 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.196662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:59445 (local address 127.6.2.194:40969)
0924 10:12:42.196918 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:42.196924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:42.197582 (+ 658us) server_negotiation.cc:407] Connection header received
0924 10:12:42.197681 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:42.197689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:42.197845 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:42.197962 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:42.198652 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.199734 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.202227 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.202951 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.203049 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.203205 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:42.203570 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 10:12:42.203710 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:12:42.204088 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.196543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.194:40969 (local address 127.6.2.193:59445)
0924 10:12:42.197340 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:42.197369 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:42.197520 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:42.198046 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:42.198051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:42.198081 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:42.198471 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:42.198480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.199904 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.199910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:42.201993 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:42.202009 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.203171 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.203181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:42.203383 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.203434 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 10:12:42.203516 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 10:12:42.203666 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":87,"thread_start_us":376,"threads_started":1}
I20260924 10:12:42.204031 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.197880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.2.193:59883 (local address 127.6.2.195:33661)
0924 10:12:42.198095 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:42.198103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:42.198210 (+ 107us) server_negotiation.cc:407] Connection header received
0924 10:12:42.198401 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:42.198405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:42.198469 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:42.198577 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:12:42.199161 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.200195 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.202195 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.203113 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.203246 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.203357 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:42.203663 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 10:12:42.203795 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 10:12:42.204274 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.197469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.195:33661 (local address 127.6.2.193:59883)
0924 10:12:42.198099 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:42.198116 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:12:42.198255 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:12:42.198599 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:12:42.198605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:12:42.198635 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:12:42.198988 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:42.198999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.200351 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.200355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:42.201984 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:12:42.201996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.203325 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.203334 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:12:42.203462 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.203484 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:12:42.203594 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 10:12:42.203714 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":22,"thread_start_us":353,"threads_started":1}
I20260924 10:12:42.205226 26330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0ad63897b242f7b00478e283017b3d" candidate_uuid: "760e23cd453046f9877517be8a8f53fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e7398c7ed447bbb3edb12a92450398" is_pre_election: true
I20260924 10:12:42.205353 26460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0ad63897b242f7b00478e283017b3d" candidate_uuid: "760e23cd453046f9877517be8a8f53fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3124c0d368642ea9667fda9312607c3" is_pre_election: true
I20260924 10:12:42.205789 26330 raft_consensus.cc:2454] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 760e23cd453046f9877517be8a8f53fa in term 0.
I20260924 10:12:42.205941 26460 raft_consensus.cc:2454] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 760e23cd453046f9877517be8a8f53fa in term 0.
I20260924 10:12:42.206563 26134 leader_election.cc:302] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 760e23cd453046f9877517be8a8f53fa, a9e7398c7ed447bbb3edb12a92450398; no voters:
I20260924 10:12:42.206957 26528 raft_consensus.cc:2781] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:12:42.207152 26528 raft_consensus.cc:482] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:12:42.207273 26528 raft_consensus.cc:3037] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:42.209616 26528 raft_consensus.cc:504] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e7398c7ed447bbb3edb12a92450398" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40969 } } peers { permanent_uuid: "760e23cd453046f9877517be8a8f53fa" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42973 } } peers { permanent_uuid: "d3124c0d368642ea9667fda9312607c3" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 33661 } }
I20260924 10:12:42.210441 26528 leader_election.cc:288] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [CANDIDATE]: Term 1 election: Requested vote from peers a9e7398c7ed447bbb3edb12a92450398 (127.6.2.194:40969), d3124c0d368642ea9667fda9312607c3 (127.6.2.195:33661)
I20260924 10:12:42.210904 26330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0ad63897b242f7b00478e283017b3d" candidate_uuid: "760e23cd453046f9877517be8a8f53fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e7398c7ed447bbb3edb12a92450398"
I20260924 10:12:42.211043 26460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0ad63897b242f7b00478e283017b3d" candidate_uuid: "760e23cd453046f9877517be8a8f53fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3124c0d368642ea9667fda9312607c3"
I20260924 10:12:42.211151 26330 raft_consensus.cc:3037] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:42.211292 26460 raft_consensus.cc:3037] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:42.213495 26460 raft_consensus.cc:2454] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 760e23cd453046f9877517be8a8f53fa in term 1.
I20260924 10:12:42.213572 26330 raft_consensus.cc:2454] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 760e23cd453046f9877517be8a8f53fa in term 1.
I20260924 10:12:42.213943 26134 leader_election.cc:302] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 760e23cd453046f9877517be8a8f53fa, d3124c0d368642ea9667fda9312607c3; no voters:
I20260924 10:12:42.214265 26528 raft_consensus.cc:2781] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:42.214715 26528 raft_consensus.cc:686] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [term 1 LEADER]: Becoming Leader. State: Replica: 760e23cd453046f9877517be8a8f53fa, State: Running, Role: LEADER
I20260924 10:12:42.215282 26528 consensus_queue.cc:237] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e7398c7ed447bbb3edb12a92450398" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40969 } } peers { permanent_uuid: "760e23cd453046f9877517be8a8f53fa" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42973 } } peers { permanent_uuid: "d3124c0d368642ea9667fda9312607c3" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 33661 } }
I20260924 10:12:42.220108 26058 catalog_manager.cc:5495] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 760e23cd453046f9877517be8a8f53fa (127.6.2.193). New cstate: current_term: 1 leader_uuid: "760e23cd453046f9877517be8a8f53fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e7398c7ed447bbb3edb12a92450398" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "760e23cd453046f9877517be8a8f53fa" member_type: VOTER last_known_addr { host: "127.6.2.193" port: 42973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3124c0d368642ea9667fda9312607c3" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 33661 } health_report { overall_health: UNKNOWN } } }
I20260924 10:12:42.256238 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.250794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.6.2.193:42973)
0924 10:12:42.251104 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:42.251111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:42.251132 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:12:42.251266 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:42.251270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:42.251378 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:42.251537 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:12:42.251926 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.252688 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.254095 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.254515 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.254654 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.255158 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:12:42.255662 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:12:42.255822 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:42.255909 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 10:12:42.255986 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 10:12:42.264864 26330 raft_consensus.cc:1261] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398 [term 1 FOLLOWER]: Refusing update from remote peer 760e23cd453046f9877517be8a8f53fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:12:42.264887 26460 raft_consensus.cc:1261] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3 [term 1 FOLLOWER]: Refusing update from remote peer 760e23cd453046f9877517be8a8f53fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:12:42.265883 26534 consensus_queue.cc:1035] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3124c0d368642ea9667fda9312607c3" member_type: VOTER last_known_addr { host: "127.6.2.195" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:12:42.266304 26528 consensus_queue.cc:1035] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e7398c7ed447bbb3edb12a92450398" member_type: VOTER last_known_addr { host: "127.6.2.194" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:12:42.293128 26057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:12:42.293490 26057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:12:42.297948 26057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 011788c98de74b7b8b099776a0e45f24: Sending DeleteTablet for 3 replicas of tablet 2f0ad63897b242f7b00478e283017b3d
I20260924 10:12:42.299057 26310 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0ad63897b242f7b00478e283017b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:42 UTC) from {username='slave'} at 127.0.0.1:54054
I20260924 10:12:42.299172 26179 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0ad63897b242f7b00478e283017b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:42 UTC) from {username='slave'} at 127.0.0.1:48336
I20260924 10:12:42.300355 26440 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0ad63897b242f7b00478e283017b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:12:42 UTC) from {username='slave'} at 127.0.0.1:46152
I20260924 10:12:42.301443 26549 tablet_replica.cc:330] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3: stopping tablet replica
I20260924 10:12:42.301956 26549 raft_consensus.cc:2229] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:12:42.302464 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 26117
I20260924 10:12:42.302637 26549 raft_consensus.cc:2258] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:42.304792 26549 ts_tablet_manager.cc:1905] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:42.313027 26549 ts_tablet_manager.cc:1918] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:42.313308 26549 log.cc:1196] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-2/wal/wals/2f0ad63897b242f7b00478e283017b3d
I20260924 10:12:42.313678 26547 tablet_replica.cc:330] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398: stopping tablet replica
I20260924 10:12:42.314095 26549 ts_tablet_manager.cc:1939] T 2f0ad63897b242f7b00478e283017b3d P d3124c0d368642ea9667fda9312607c3: Deleting consensus metadata
I20260924 10:12:42.314430 26548 tablet_replica.cc:330] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa: stopping tablet replica
I20260924 10:12:42.314993 26548 raft_consensus.cc:2229] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:12:42.315816 26548 raft_consensus.cc:2258] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:42.316424 26044 catalog_manager.cc:4841] TS d3124c0d368642ea9667fda9312607c3 (127.6.2.195:33661): tablet 2f0ad63897b242f7b00478e283017b3d (table test-table [id=fe49ac56cd544217a040f1e57ee556ac]) successfully deleted
I20260924 10:12:42.316481 26547 raft_consensus.cc:2229] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:12:42.317040 26547 raft_consensus.cc:2258] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:12:42.319633 26547 ts_tablet_manager.cc:1905] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:42.325111 26548 ts_tablet_manager.cc:1905] T 2f0ad63897b242f7b00478e283017b3d P 760e23cd453046f9877517be8a8f53fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:12:42.325672 26547 ts_tablet_manager.cc:1918] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:12:42.325834 26547 log.cc:1196] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398: Deleting WAL directory at /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790244437048964-6155-0/minicluster-data/ts-1/wal/wals/2f0ad63897b242f7b00478e283017b3d
I20260924 10:12:42.326356 26547 ts_tablet_manager.cc:1939] T 2f0ad63897b242f7b00478e283017b3d P a9e7398c7ed447bbb3edb12a92450398: Deleting consensus metadata
I20260924 10:12:42.328191 26044 catalog_manager.cc:4841] TS a9e7398c7ed447bbb3edb12a92450398 (127.6.2.194:40969): tablet 2f0ad63897b242f7b00478e283017b3d (table test-table [id=fe49ac56cd544217a040f1e57ee556ac]) successfully deleted
I20260924 10:12:42.328677 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.304157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.6.2.193:42973)
0924 10:12:42.304409 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:42.304418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:42.319505 (+ 15087us) server_negotiation.cc:407] Connection header received
0924 10:12:42.319753 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:42.319760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:42.319867 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:42.320004 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:42.320484 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.321827 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.323708 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.324488 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.324645 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.326437 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:42.326450 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:42.326478 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:42.327450 (+ 972us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:42.327525 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:42.327569 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:42.327656 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:42.328033 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 10:12:42.328187 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:12:42.329892 26239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:42.427960 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 26117
W20260924 10:12:42.451218 26043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.2.193:42973 (error 108)
I20260924 10:12:42.451360 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 26248
I20260924 10:12:42.453433 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.452863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42973 (local address 127.0.0.1:48374)
0924 10:12:42.453091 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:42.453230 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 10:12:42.453729 26043 catalog_manager.cc:4584] TS 760e23cd453046f9877517be8a8f53fa (127.6.2.193:42973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f0ad63897b242f7b00478e283017b3d: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42973: connect: Connection refused (error 111)
I20260924 10:12:42.459868 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.452673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.6.2.194:40969)
0924 10:12:42.452866 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:42.452870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:42.452988 (+ 118us) server_negotiation.cc:407] Connection header received
0924 10:12:42.453214 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:42.453221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:42.453332 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:42.453458 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:42.454189 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.455122 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.456589 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.457044 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.457193 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.458383 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:42.458393 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:42.458409 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:42.458998 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:42.459045 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:42.459116 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:42.459256 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:42.459407 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 10:12:42.459522 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:12:42.460592 26370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:42.507346 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.506840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42973 (local address 127.0.0.1:48382)
0924 10:12:42.507079 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:42.507180 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 10:12:42.508432 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.508078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42973 (local address 127.0.0.1:48394)
0924 10:12:42.508222 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:42.508314 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 10:12:42.550266 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 26248
I20260924 10:12:42.567884 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 26378
I20260924 10:12:42.569636 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.569098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42973 (local address 127.0.0.1:48406)
0924 10:12:42.569299 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:42.569450 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 10:12:42.570719 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.570344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42973 (local address 127.0.0.1:48412)
0924 10:12:42.570501 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:42.570582 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 10:12:42.574834 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.568639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.6.2.195:33661)
0924 10:12:42.568800 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:42.568805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:42.568825 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:12:42.569019 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:42.569023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:42.569121 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:42.569257 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:42.569647 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.570657 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.572074 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:42.572449 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:42.572549 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:42.573599 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:42.573607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:42.573623 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:42.574213 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:42.574263 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:42.574283 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:42.574371 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:42.574520 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 10:12:42.574608 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:12:42.575601 26500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:42.651458 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.650863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42973 (local address 127.0.0.1:48414)
0924 10:12:42.651183 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:42.651283 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 10:12:42.652667 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:42.652250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.193:42973 (local address 127.0.0.1:48424)
0924 10:12:42.652470 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 10:12:42.652552 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.193:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 10:12:42.664975 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 26378
I20260924 10:12:42.682003 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 26027
I20260924 10:12:42.683248 26088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:42.793361 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 26027
2026-09-24T10:12:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2752 ms)
[----------] 6 tests from EncryptionPolicyTest (21640 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:12:42 dist-test-slave-s3n3 krb5kdc[26558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:12:42 dist-test-slave-s3n3 krb5kdc[26558](info): set up 2 sockets
Sep 24 10:12:42 dist-test-slave-s3n3 krb5kdc[26558](info): commencing operation
krb5kdc: starting...
W20260924 10:12:43.923593 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:12:44 dist-test-slave-s3n3 krb5kdc[26558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:12:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:12:44.178887 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38013
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:44.506036 26571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:44.506323 26571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:44.506418 26571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:44.513650 26571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:44.513767 26571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:44.513814 26571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:44.513868 26571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:44.513926 26571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:44.524207 26571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38013
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:44.525545 26571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:44.526851 26571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:44.541007 26576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:44.541072 26579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:44.541018 26577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:44.547451 26571 server_base.cc:1034] running on GCE node
I20260924 10:12:44.548343 26571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:44.549830 26571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:44.551134 26571 hybrid_clock.cc:648] HybridClock initialized: now 1790244764550985 us; error 75 us; skew 500 ppm
I20260924 10:12:44.551491 26571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:44 dist-test-slave-s3n3 krb5kdc[26558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:44.560449 26571 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:12:44.562150 26571 webserver.cc:460] Webserver started at http://127.0.0.1:44043/ using document root <none> and password file <none>
I20260924 10:12:44.562914 26571 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:44.563102 26571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:44.563418 26571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:44.566432 26571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "ed517dcbc35a4565b2474ae34462261f"
format_stamp: "Formatted at 2026-09-24 10:12:44 on dist-test-slave-s3n3"
server_key: "8b8c0c7e325bd3123f2b833b2451915c"
server_key_iv: "446451c19afaaf9c4b865d9a8277fc39"
server_key_version: "encryptionkey@0"
I20260924 10:12:44.567301 26571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "ed517dcbc35a4565b2474ae34462261f"
format_stamp: "Formatted at 2026-09-24 10:12:44 on dist-test-slave-s3n3"
server_key: "8b8c0c7e325bd3123f2b833b2451915c"
server_key_iv: "446451c19afaaf9c4b865d9a8277fc39"
server_key_version: "encryptionkey@0"
I20260924 10:12:44.572504 26571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 10:12:44.575985 26586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:44.577231 26571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:12:44.577425 26571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "ed517dcbc35a4565b2474ae34462261f"
format_stamp: "Formatted at 2026-09-24 10:12:44 on dist-test-slave-s3n3"
server_key: "8b8c0c7e325bd3123f2b833b2451915c"
server_key_iv: "446451c19afaaf9c4b865d9a8277fc39"
server_key_version: "encryptionkey@0"
I20260924 10:12:44.577620 26571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:44.597749 26571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:44.601859 26571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:44.602144 26571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:44.620443 26571 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38013
I20260924 10:12:44.620466 26637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38013 every 8 connection(s)
I20260924 10:12:44.621891 26571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:44.623932 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 26571
I20260924 10:12:44.624140 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:44.624950 6155 external_mini_cluster.cc:1428] Setting key a1a626541871f9381501a9110e7bbb76
I20260924 10:12:44.626322 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 10:12:44 dist-test-slave-s3n3 krb5kdc[26558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:12:44.640591 26638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f: Bootstrap starting.
I20260924 10:12:44.643836 26638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:44.645198 26638 log.cc:824] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:44.648416 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:44.626942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48675 (local address 127.0.0.1:38013)
0924 10:12:44.627764 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:44.627775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:44.627814 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:12:44.629344 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:44.629364 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:44.629749 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:44.630054 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:44.631838 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:44.633168 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:44.635024 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:44.635717 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:44.635860 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:44.640743 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:44.640754 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:44.640772 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:44.640829 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:44.643869 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:44.644855 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:44.644861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:44.644866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:44.645026 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:44.645557 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:44.645561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:44.645563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:44.646302 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:44.646654 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:44.647024 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 10:12:44.647401 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":371,"threads_started":1}
I20260924 10:12:44.648607 26638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f: No bootstrap required, opened a new log
I20260924 10:12:44.652566 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:12:44.659771 26638 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed517dcbc35a4565b2474ae34462261f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38013 } }
I20260924 10:12:44.660144 26638 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:44.660244 26638 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed517dcbc35a4565b2474ae34462261f, State: Initialized, Role: FOLLOWER
I20260924 10:12:44.660812 26638 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed517dcbc35a4565b2474ae34462261f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38013 } }
I20260924 10:12:44.661089 26638 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:44.661252 26638 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:44.661435 26638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:44.663962 26638 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed517dcbc35a4565b2474ae34462261f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38013 } }
I20260924 10:12:44.664580 26638 leader_election.cc:302] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed517dcbc35a4565b2474ae34462261f; no voters:
I20260924 10:12:44.665278 26638 leader_election.cc:288] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:44.665423 26648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:44.667351 26648 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [term 1 LEADER]: Becoming Leader. State: Replica: ed517dcbc35a4565b2474ae34462261f, State: Running, Role: LEADER
I20260924 10:12:44.667428 26638 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:44.668018 26648 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed517dcbc35a4565b2474ae34462261f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38013 } }
I20260924 10:12:44.671424 26649 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed517dcbc35a4565b2474ae34462261f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed517dcbc35a4565b2474ae34462261f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38013 } } }
I20260924 10:12:44.671875 26649 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:44.673130 26656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:44.675051 26650 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed517dcbc35a4565b2474ae34462261f. Latest consensus state: current_term: 1 leader_uuid: "ed517dcbc35a4565b2474ae34462261f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed517dcbc35a4565b2474ae34462261f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38013 } } }
I20260924 10:12:44.675434 26650 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:44.679549 26656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:44.680073 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:44.658429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54635 (local address 127.0.0.1:38013)
0924 10:12:44.658627 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:44.658634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:44.659012 (+ 378us) server_negotiation.cc:407] Connection header received
0924 10:12:44.659205 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:44.659225 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:44.659301 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:44.659412 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:44.660998 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:44.662190 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:44.663781 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:44.664409 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:44.664578 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:44.668036 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:44.668042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:44.668047 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:44.668085 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:44.674513 (+ 6428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:44.675851 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:44.675860 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:44.675866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:44.676129 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:44.676894 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:44.676902 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:44.676908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:44.677443 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:44.679015 (+ 1572us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:44.679301 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 10:12:44.679557 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:12:44.690500 26656 catalog_manager.cc:1347] Generated new cluster ID: ee6e27520a984c64856922b21f106a44
I20260924 10:12:44.690616 26656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:44.728042 26656 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:44.729355 26656 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:44.736052 26656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed517dcbc35a4565b2474ae34462261f: Generated new TSK 0
I20260924 10:12:44.736876 26656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:12:44.753625 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 26571
I20260924 10:12:44.755029 26633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:44.855463 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 26571
W20260924 10:12:44.873529 26556 connection.cc:535] client connection to 127.0.0.1:38013 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38013): Connection reset by peer (error 104)
2026-09-24T10:12:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2053 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:12:44 dist-test-slave-s3n3 krb5kdc[26670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:12:44 dist-test-slave-s3n3 krb5kdc[26670](info): set up 2 sockets
Sep 24 10:12:44 dist-test-slave-s3n3 krb5kdc[26670](info): commencing operation
krb5kdc: starting...
W20260924 10:12:45.959457 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:12:46 dist-test-slave-s3n3 krb5kdc[26670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:12:46.210628 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:46.532428 26683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:46.532770 26683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:46.532941 26683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:46.540158 26683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:46.540315 26683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:46.540421 26683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:46.540532 26683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:46.540606 26683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:46.550223 26683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:46.551586 26683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:46.552942 26683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:46.564556 26688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:46.564577 26691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:46.564574 26689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:46.565764 26683 server_base.cc:1034] running on GCE node
I20260924 10:12:46.566576 26683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:46.568041 26683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:46.569280 26683 hybrid_clock.cc:648] HybridClock initialized: now 1790244766569240 us; error 51 us; skew 500 ppm
I20260924 10:12:46.569643 26683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:46 dist-test-slave-s3n3 krb5kdc[26670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:46.573033 26683 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:12:46.574502 26683 webserver.cc:460] Webserver started at http://127.0.0.1:41437/ using document root <none> and password file <none>
I20260924 10:12:46.575227 26683 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:46.575330 26683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:46.575579 26683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:46.578527 26683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "12ba80839cdb4b55a4521d5b2f2e8ff3"
format_stamp: "Formatted at 2026-09-24 10:12:46 on dist-test-slave-s3n3"
server_key: "a68d7b9ea6dc4193a6bb15c979741015"
server_key_iv: "f24b2d1e2e02d49859164571d91bcaa3"
server_key_version: "encryptionkey@0"
I20260924 10:12:46.579316 26683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "12ba80839cdb4b55a4521d5b2f2e8ff3"
format_stamp: "Formatted at 2026-09-24 10:12:46 on dist-test-slave-s3n3"
server_key: "a68d7b9ea6dc4193a6bb15c979741015"
server_key_iv: "f24b2d1e2e02d49859164571d91bcaa3"
server_key_version: "encryptionkey@0"
I20260924 10:12:46.584403 26683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 10:12:46.587800 26698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:46.589179 26683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:12:46.589382 26683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "12ba80839cdb4b55a4521d5b2f2e8ff3"
format_stamp: "Formatted at 2026-09-24 10:12:46 on dist-test-slave-s3n3"
server_key: "a68d7b9ea6dc4193a6bb15c979741015"
server_key_iv: "f24b2d1e2e02d49859164571d91bcaa3"
server_key_version: "encryptionkey@0"
I20260924 10:12:46.589583 26683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:46.613119 26683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:46.617519 26683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:46.617893 26683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:46.636889 26683 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41531
I20260924 10:12:46.636994 26749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41531 every 8 connection(s)
I20260924 10:12:46.638300 26683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:46.641932 26750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:46.644040 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 26683
I20260924 10:12:46.644212 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:46.644784 6155 external_mini_cluster.cc:1428] Setting key 8ca751b48cf66bb98c913fe3535e3a3f
I20260924 10:12:46.656139 26750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3: Bootstrap starting.
I20260924 10:12:46.657990 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:46.646589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51315 (local address 127.0.0.1:41531)
0924 10:12:46.647456 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:46.647466 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:46.647505 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:12:46.649075 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:46.649095 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:46.649479 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:46.649771 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:46.650309 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:46.651608 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:46.653521 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:46.654190 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:46.654328 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:46.655722 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:46.655730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:46.655743 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:46.656348 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:46.656430 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:46.656456 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:46.656578 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:46.656845 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 10:12:46.657029 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":428,"threads_started":1}
I20260924 10:12:46.659379 26750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:46.660461 26750 log.cc:824] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:46.661625 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:12:46.663204 26750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3: No bootstrap required, opened a new log
I20260924 10:12:46.671932 26750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ba80839cdb4b55a4521d5b2f2e8ff3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260924 10:12:46.672410 26750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:46.672528 26750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ba80839cdb4b55a4521d5b2f2e8ff3, State: Initialized, Role: FOLLOWER
I20260924 10:12:46.672591 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:46.666078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59679 (local address 127.0.0.1:41531)
0924 10:12:46.666256 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:46.666262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:46.666579 (+ 317us) server_negotiation.cc:407] Connection header received
0924 10:12:46.666726 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:46.666731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:46.666809 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:46.666917 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:46.667434 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:46.668601 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:46.669893 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:46.670592 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:46.670719 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:46.671813 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:46.671819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:46.671823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:46.671841 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:46.671901 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:46.671929 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:46.672050 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:46.672184 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 10:12:46.672240 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:12:46.673207 26750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ba80839cdb4b55a4521d5b2f2e8ff3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260924 10:12:46.673453 26750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:46.673609 26750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:46.673789 26750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:46.675976 26750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ba80839cdb4b55a4521d5b2f2e8ff3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260924 10:12:46.676450 26750 leader_election.cc:302] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12ba80839cdb4b55a4521d5b2f2e8ff3; no voters:
I20260924 10:12:46.677019 26750 leader_election.cc:288] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:46.677162 26760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:46.678290 26760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [term 1 LEADER]: Becoming Leader. State: Replica: 12ba80839cdb4b55a4521d5b2f2e8ff3, State: Running, Role: LEADER
I20260924 10:12:46.678921 26760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ba80839cdb4b55a4521d5b2f2e8ff3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260924 10:12:46.679098 26750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:46.682221 26762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12ba80839cdb4b55a4521d5b2f2e8ff3. Latest consensus state: current_term: 1 leader_uuid: "12ba80839cdb4b55a4521d5b2f2e8ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ba80839cdb4b55a4521d5b2f2e8ff3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } } }
I20260924 10:12:46.682192 26761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12ba80839cdb4b55a4521d5b2f2e8ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ba80839cdb4b55a4521d5b2f2e8ff3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } } }
I20260924 10:12:46.682678 26762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:46.682694 26761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:46.684749 26767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:46.689431 26767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:46.698402 26767 catalog_manager.cc:1347] Generated new cluster ID: 201c520afc394d6a8c128ef1b1ceb96b
I20260924 10:12:46.698537 26767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:46.713498 26767 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:46.714807 26767 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:46.730202 26767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12ba80839cdb4b55a4521d5b2f2e8ff3: Generated new TSK 0
I20260924 10:12:46.731235 26767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:12:46.748489 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 26683
W20260924 10:12:46.749804 26745 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51315
W20260924 10:12:46.750669 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 26683: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:12:46.750854 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 26683
W20260924 10:12:46.768620 26668 connection.cc:535] client connection to 127.0.0.1:41531 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41531): Connection reset by peer (error 104)
2026-09-24T10:12:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1895 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:12:46 dist-test-slave-s3n3 krb5kdc[26781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:12:46 dist-test-slave-s3n3 krb5kdc[26781](info): set up 2 sockets
Sep 24 10:12:46 dist-test-slave-s3n3 krb5kdc[26781](info): commencing operation
krb5kdc: starting...
W20260924 10:12:47.875494 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:12:48 dist-test-slave-s3n3 krb5kdc[26781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:12:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:12:48.130596 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:48.465092 26794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:48.465382 26794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:48.465476 26794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:48.472764 26794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:48.472890 26794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:48.472937 26794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:48.472997 26794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:48.473034 26794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:48.483091 26794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:48.484372 26794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:48.485745 26794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:48.497709 26799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:48.497785 26802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:48.497805 26800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:48.498697 26794 server_base.cc:1034] running on GCE node
I20260924 10:12:48.499521 26794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:48.500933 26794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:48.502187 26794 hybrid_clock.cc:648] HybridClock initialized: now 1790244768502140 us; error 41 us; skew 500 ppm
I20260924 10:12:48.502578 26794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:48 dist-test-slave-s3n3 krb5kdc[26781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:48.505843 26794 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:12:48.507391 26794 webserver.cc:460] Webserver started at http://127.0.0.1:42433/ using document root <none> and password file <none>
I20260924 10:12:48.508080 26794 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:48.508230 26794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:48.508546 26794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:48.511543 26794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "a07f3b56083246cd9590ae9480114406"
format_stamp: "Formatted at 2026-09-24 10:12:48 on dist-test-slave-s3n3"
server_key: "85656adcf55effeb4326fd6929d981e1"
server_key_iv: "18348bcaefc3b960442b20ffaa0ff464"
server_key_version: "encryptionkey@0"
I20260924 10:12:48.512305 26794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "a07f3b56083246cd9590ae9480114406"
format_stamp: "Formatted at 2026-09-24 10:12:48 on dist-test-slave-s3n3"
server_key: "85656adcf55effeb4326fd6929d981e1"
server_key_iv: "18348bcaefc3b960442b20ffaa0ff464"
server_key_version: "encryptionkey@0"
I20260924 10:12:48.517445 26794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:12:48.520965 26809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:48.522176 26794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:12:48.522365 26794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "a07f3b56083246cd9590ae9480114406"
format_stamp: "Formatted at 2026-09-24 10:12:48 on dist-test-slave-s3n3"
server_key: "85656adcf55effeb4326fd6929d981e1"
server_key_iv: "18348bcaefc3b960442b20ffaa0ff464"
server_key_version: "encryptionkey@0"
I20260924 10:12:48.522567 26794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:48.528167 26794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:48.532192 26794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:48.532498 26794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:48.550514 26794 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42535
I20260924 10:12:48.550597 26860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42535 every 8 connection(s)
I20260924 10:12:48.551928 26794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:48.555841 26861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:48.556164 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 26794
I20260924 10:12:48.556350 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:48.556916 6155 external_mini_cluster.cc:1428] Setting key af4f40f6df74d5c1690cd74303f3abcb
I20260924 10:12:48.565130 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:48.558687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37383 (local address 127.0.0.1:42535)
0924 10:12:48.559590 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:48.559604 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:48.559653 (+ 49us) server_negotiation.cc:407] Connection header received
0924 10:12:48.561266 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:48.561311 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:48.561748 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:48.562206 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:48.563237 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:48.563245 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:48.563261 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:48.563295 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:48.563365 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:48.563393 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:48.563625 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:48.563760 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 10:12:48.563953 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":358,"threads_started":1}
I20260924 10:12:48.568885 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:12:48.570417 26861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406: Bootstrap starting.
I20260924 10:12:48.573899 26861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:48.575170 26861 log.cc:824] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:48.576013 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:48.573261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47879 (local address 127.0.0.1:42535)
0924 10:12:48.573452 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:48.573460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:48.573777 (+ 317us) server_negotiation.cc:407] Connection header received
0924 10:12:48.574028 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:48.574037 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:48.574085 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:48.574193 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:48.575282 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:48.575305 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:48.575312 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:48.575334 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:48.575387 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:48.575403 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:48.575531 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:48.575642 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 10:12:48.575702 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:12:48.578157 26861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406: No bootstrap required, opened a new log
I20260924 10:12:48.585846 26861 raft_consensus.cc:348] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07f3b56083246cd9590ae9480114406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260924 10:12:48.586221 26861 raft_consensus.cc:374] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:48.586321 26861 raft_consensus.cc:729] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07f3b56083246cd9590ae9480114406, State: Initialized, Role: FOLLOWER
I20260924 10:12:48.587013 26861 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07f3b56083246cd9590ae9480114406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260924 10:12:48.587298 26861 raft_consensus.cc:388] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:48.587456 26861 raft_consensus.cc:482] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:48.587623 26861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:48.590088 26861 raft_consensus.cc:504] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07f3b56083246cd9590ae9480114406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260924 10:12:48.590601 26861 leader_election.cc:302] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07f3b56083246cd9590ae9480114406; no voters:
I20260924 10:12:48.591270 26861 leader_election.cc:288] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:48.591411 26871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:48.592083 26871 raft_consensus.cc:686] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [term 1 LEADER]: Becoming Leader. State: Replica: a07f3b56083246cd9590ae9480114406, State: Running, Role: LEADER
I20260924 10:12:48.592677 26871 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07f3b56083246cd9590ae9480114406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260924 10:12:48.593369 26861 sys_catalog.cc:564] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:48.595880 26873 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07f3b56083246cd9590ae9480114406. Latest consensus state: current_term: 1 leader_uuid: "a07f3b56083246cd9590ae9480114406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07f3b56083246cd9590ae9480114406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } } }
I20260924 10:12:48.596282 26873 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:48.596026 26872 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a07f3b56083246cd9590ae9480114406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07f3b56083246cd9590ae9480114406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } } }
I20260924 10:12:48.596628 26872 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:48.598687 26878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:48.603765 26878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:48.612124 26878 catalog_manager.cc:1347] Generated new cluster ID: f2098e20784c45c69c264a207f58a570
I20260924 10:12:48.612244 26878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:48.633865 26878 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:48.634320 26878 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:48.644002 26878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a07f3b56083246cd9590ae9480114406: Generated new TSK 0
I20260924 10:12:48.644817 26878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:12:48.661520 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 26794
W20260924 10:12:48.662632 26856 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37383
W20260924 10:12:48.663335 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 26794: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:12:48.663499 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 26794
2026-09-24T10:12:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:12:48 dist-test-slave-s3n3 krb5kdc[26892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:12:48 dist-test-slave-s3n3 krb5kdc[26892](info): set up 2 sockets
Sep 24 10:12:48 dist-test-slave-s3n3 krb5kdc[26892](info): commencing operation
krb5kdc: starting...
W20260924 10:12:49.772053 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:12:49 dist-test-slave-s3n3 krb5kdc[26892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:12:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:12:50.030684 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:50.361140 26905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:50.361454 26905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:50.361621 26905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:50.368947 26905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:50.369069 26905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:50.369166 26905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:50.369277 26905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:50.369335 26905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:50.379360 26905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:50.380702 26905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:50.382010 26905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:50.393631 26910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:50.393671 26911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:50.393697 26913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:50.394435 26905 server_base.cc:1034] running on GCE node
I20260924 10:12:50.395238 26905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:50.396656 26905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:50.397881 26905 hybrid_clock.cc:648] HybridClock initialized: now 1790244770397846 us; error 63 us; skew 500 ppm
I20260924 10:12:50.398226 26905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:50 dist-test-slave-s3n3 krb5kdc[26892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:50.407269 26905 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:12:50.408830 26905 webserver.cc:460] Webserver started at http://127.0.0.1:46235/ using document root <none> and password file <none>
I20260924 10:12:50.409559 26905 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:50.409699 26905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:50.409997 26905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:50.413020 26905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "7b8d6bf0f952468d82df2fbc6d890162"
format_stamp: "Formatted at 2026-09-24 10:12:50 on dist-test-slave-s3n3"
server_key: "e41954d97aea0a835713c720040da119"
server_key_iv: "6b54cb6d3015548c0aa1d7b9f5ce1507"
server_key_version: "encryptionkey@0"
I20260924 10:12:50.413798 26905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "7b8d6bf0f952468d82df2fbc6d890162"
format_stamp: "Formatted at 2026-09-24 10:12:50 on dist-test-slave-s3n3"
server_key: "e41954d97aea0a835713c720040da119"
server_key_iv: "6b54cb6d3015548c0aa1d7b9f5ce1507"
server_key_version: "encryptionkey@0"
I20260924 10:12:50.418859 26905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:12:50.422185 26920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:50.423642 26905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:12:50.423825 26905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "7b8d6bf0f952468d82df2fbc6d890162"
format_stamp: "Formatted at 2026-09-24 10:12:50 on dist-test-slave-s3n3"
server_key: "e41954d97aea0a835713c720040da119"
server_key_iv: "6b54cb6d3015548c0aa1d7b9f5ce1507"
server_key_version: "encryptionkey@0"
I20260924 10:12:50.424021 26905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:50.436645 26905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:50.440608 26905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:50.440886 26905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:50.459568 26905 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38235
I20260924 10:12:50.459587 26971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38235 every 8 connection(s)
I20260924 10:12:50.460991 26905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:50.464411 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 26905
I20260924 10:12:50.464602 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:50.465157 26972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:50.465205 6155 external_mini_cluster.cc:1428] Setting key ce337ef350c020a97d39ed0a2e278b33
I20260924 10:12:50.479702 26972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162: Bootstrap starting.
Sep 24 10:12:50 dist-test-slave-s3n3 krb5kdc[26892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:12:50.483695 26972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:50.485033 26972 log.cc:824] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:50.488620 26972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162: No bootstrap required, opened a new log
I20260924 10:12:50.490038 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:50.466889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57047 (local address 127.0.0.1:38235)
0924 10:12:50.467715 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:50.467725 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:50.467766 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:12:50.469737 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:50.469757 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:50.470166 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:50.470488 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:50.472341 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:50.473609 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:50.475383 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:50.476069 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:50.476202 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:50.481027 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:50.481039 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:50.481066 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:50.481136 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:50.485275 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:50.486208 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:50.486214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:50.486220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:50.486534 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:50.486935 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:50.486939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:50.486942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:50.487667 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:50.487969 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:50.488297 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 10:12:50.488729 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":394,"threads_started":1}
I20260924 10:12:50.493719 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:12:50.497547 26972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8d6bf0f952468d82df2fbc6d890162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260924 10:12:50.498028 26972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:50.498189 26972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8d6bf0f952468d82df2fbc6d890162, State: Initialized, Role: FOLLOWER
I20260924 10:12:50.498812 26972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8d6bf0f952468d82df2fbc6d890162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260924 10:12:50.499125 26972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:50.499271 26972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:50.499446 26972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:50.501674 26972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8d6bf0f952468d82df2fbc6d890162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260924 10:12:50.502159 26972 leader_election.cc:302] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b8d6bf0f952468d82df2fbc6d890162; no voters:
I20260924 10:12:50.502799 26972 leader_election.cc:288] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:50.502979 26982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:50.503720 26982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8d6bf0f952468d82df2fbc6d890162, State: Running, Role: LEADER
I20260924 10:12:50.504496 26982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8d6bf0f952468d82df2fbc6d890162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260924 10:12:50.507369 26982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b8d6bf0f952468d82df2fbc6d890162. Latest consensus state: current_term: 1 leader_uuid: "7b8d6bf0f952468d82df2fbc6d890162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8d6bf0f952468d82df2fbc6d890162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } } }
I20260924 10:12:50.507701 26982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:50.508409 26984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b8d6bf0f952468d82df2fbc6d890162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8d6bf0f952468d82df2fbc6d890162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } } }
I20260924 10:12:50.508864 26984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:50.510795 26972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:50.512552 26986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:50.515183 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:50.497991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34951 (local address 127.0.0.1:38235)
0924 10:12:50.498181 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:50.498189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:50.498587 (+ 398us) server_negotiation.cc:407] Connection header received
0924 10:12:50.498769 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:50.498774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:50.498836 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:50.498934 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:50.500424 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:50.501134 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:50.502436 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:50.502996 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:50.503177 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:50.505840 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:50.505845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:50.505848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:50.505877 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:50.510674 (+ 4797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:50.511639 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:50.511647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:50.511653 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:50.511809 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:50.512270 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:50.512275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:50.512279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:50.512765 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:50.513055 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:50.514331 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0924 10:12:50.514607 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:12:50.519450 26986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:50.528080 26986 catalog_manager.cc:1347] Generated new cluster ID: f31dd134216247d08ce68982302fe2f3
I20260924 10:12:50.528229 26986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:50.539994 26986 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:50.541224 26986 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:50.553833 26986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b8d6bf0f952468d82df2fbc6d890162: Generated new TSK 0
I20260924 10:12:50.554977 26986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:12:50.570439 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 26905
I20260924 10:12:50.571825 26967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:50.664932 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 26905
2026-09-24T10:12:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2004 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:12:50 dist-test-slave-s3n3 krb5kdc[27004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:12:50 dist-test-slave-s3n3 krb5kdc[27004](info): set up 2 sockets
Sep 24 10:12:50 dist-test-slave-s3n3 krb5kdc[27004](info): commencing operation
krb5kdc: starting...
W20260924 10:12:51.771426 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:12:51 dist-test-slave-s3n3 krb5kdc[27004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:12:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:12:52.018278 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:52.347982 27017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:52.348246 27017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:52.348413 27017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:52.355638 27017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:52.355736 27017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:52.355841 27017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:52.355934 27017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:52.356005 27017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:52.365615 27017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:52.366784 27017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:52.368072 27017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:52.379352 27022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:52.379398 27025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:52.379436 27023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:52.380424 27017 server_base.cc:1034] running on GCE node
I20260924 10:12:52.381217 27017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:52.382645 27017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:52.383905 27017 hybrid_clock.cc:648] HybridClock initialized: now 1790244772383869 us; error 44 us; skew 500 ppm
I20260924 10:12:52.384234 27017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:52 dist-test-slave-s3n3 krb5kdc[27004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:52.387346 27017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:12:52.388798 27017 webserver.cc:460] Webserver started at http://127.0.0.1:43979/ using document root <none> and password file <none>
I20260924 10:12:52.389459 27017 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:52.389595 27017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:52.389889 27017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:52.392815 27017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "7a88194b13794f75b0c5b537d868134c"
format_stamp: "Formatted at 2026-09-24 10:12:52 on dist-test-slave-s3n3"
server_key: "90abd01026507e31ad04363660d0f1c7"
server_key_iv: "5ac7c9706b3fbfdfa28ca50165019a51"
server_key_version: "encryptionkey@0"
I20260924 10:12:52.393574 27017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "7a88194b13794f75b0c5b537d868134c"
format_stamp: "Formatted at 2026-09-24 10:12:52 on dist-test-slave-s3n3"
server_key: "90abd01026507e31ad04363660d0f1c7"
server_key_iv: "5ac7c9706b3fbfdfa28ca50165019a51"
server_key_version: "encryptionkey@0"
I20260924 10:12:52.399362 27017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:12:52.402716 27032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:52.404140 27017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:12:52.404323 27017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "7a88194b13794f75b0c5b537d868134c"
format_stamp: "Formatted at 2026-09-24 10:12:52 on dist-test-slave-s3n3"
server_key: "90abd01026507e31ad04363660d0f1c7"
server_key_iv: "5ac7c9706b3fbfdfa28ca50165019a51"
server_key_version: "encryptionkey@0"
I20260924 10:12:52.404513 27017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:52.426882 27017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:52.430742 27017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:52.431033 27017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:52.449419 27017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45581
I20260924 10:12:52.449496 27083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45581 every 8 connection(s)
I20260924 10:12:52.450765 27017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:52.454252 27084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:52.461014 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 27017
I20260924 10:12:52.461191 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:52.461710 6155 external_mini_cluster.cc:1428] Setting key ba81fa3a0c7a541b872e1c1c4afadbed
I20260924 10:12:52.464856 27084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c: Bootstrap starting.
I20260924 10:12:52.468091 27084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:52.469194 27084 log.cc:824] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:52.472074 27084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c: No bootstrap required, opened a new log
I20260924 10:12:52.474709 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:52.463492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48701 (local address 127.0.0.1:45581)
0924 10:12:52.464235 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:52.464245 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:52.464282 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:12:52.466031 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:52.466052 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:52.466430 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:52.466729 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:52.467322 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:52.468700 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:52.470378 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:52.471044 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:52.471225 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:52.472422 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:52.472429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:52.472441 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:52.472994 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:52.473066 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:52.473092 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:52.473190 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:52.473443 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 10:12:52.473723 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":357,"threads_started":1}
I20260924 10:12:52.478513 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:12:52.481514 27084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a88194b13794f75b0c5b537d868134c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45581 } }
I20260924 10:12:52.482004 27084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:52.482133 27084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a88194b13794f75b0c5b537d868134c, State: Initialized, Role: FOLLOWER
I20260924 10:12:52.482815 27084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a88194b13794f75b0c5b537d868134c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45581 } }
I20260924 10:12:52.483114 27084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:52.483258 27084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:52.483461 27084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:52.486337 27084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a88194b13794f75b0c5b537d868134c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45581 } }
I20260924 10:12:52.486842 27084 leader_election.cc:302] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a88194b13794f75b0c5b537d868134c; no voters:
I20260924 10:12:52.487488 27084 leader_election.cc:288] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:52.487623 27094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:52.488778 27094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [term 1 LEADER]: Becoming Leader. State: Replica: 7a88194b13794f75b0c5b537d868134c, State: Running, Role: LEADER
I20260924 10:12:52.489388 27094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a88194b13794f75b0c5b537d868134c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45581 } }
I20260924 10:12:52.489751 27084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:52.491696 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:52.483914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50291 (local address 127.0.0.1:45581)
0924 10:12:52.484161 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:52.484170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:52.484695 (+ 525us) server_negotiation.cc:407] Connection header received
0924 10:12:52.484804 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:52.484809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:52.484891 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:52.484999 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:52.485465 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:52.486471 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:52.488514 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:52.489244 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:52.489381 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:52.490576 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:52.490582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:52.490587 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:52.490606 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:52.490653 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:52.490671 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:52.490766 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:52.491250 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 10:12:52.491343 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:12:52.492193 27096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a88194b13794f75b0c5b537d868134c. Latest consensus state: current_term: 1 leader_uuid: "7a88194b13794f75b0c5b537d868134c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a88194b13794f75b0c5b537d868134c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45581 } } }
I20260924 10:12:52.492605 27096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:52.495565 27095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a88194b13794f75b0c5b537d868134c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a88194b13794f75b0c5b537d868134c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45581 } } }
I20260924 10:12:52.496034 27095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:52.496845 27099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:52.501168 27099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:52.510013 27099 catalog_manager.cc:1347] Generated new cluster ID: 6bc3b9423aba4ebf8d32ff6032a66c77
I20260924 10:12:52.510119 27099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:52.526433 27099 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:52.528193 27099 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:52.542989 27099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a88194b13794f75b0c5b537d868134c: Generated new TSK 0
I20260924 10:12:52.544085 27099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:12:52.559322 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27017
W20260924 10:12:52.560343 27079 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48701
W20260924 10:12:52.560968 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27017: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:12:52.561108 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 27017
2026-09-24T10:12:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1892 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:12:52 dist-test-slave-s3n3 krb5kdc[27115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:12:52 dist-test-slave-s3n3 krb5kdc[27115](info): set up 2 sockets
Sep 24 10:12:52 dist-test-slave-s3n3 krb5kdc[27115](info): commencing operation
krb5kdc: starting...
W20260924 10:12:53.707595 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:12:53 dist-test-slave-s3n3 krb5kdc[27115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:12:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:12:53.954159 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35521
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:54.283020 27128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:54.283427 27128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:54.283583 27128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:54.290772 27128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:54.290928 27128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:54.291029 27128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:54.291148 27128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:54.291224 27128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:54.301075 27128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:54.302378 27128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:54.303694 27128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:54.315801 27133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:54.315824 27134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:54.315754 27136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:54.316574 27128 server_base.cc:1034] running on GCE node
I20260924 10:12:54.317301 27128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:54.318643 27128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:54.319862 27128 hybrid_clock.cc:648] HybridClock initialized: now 1790244774319813 us; error 45 us; skew 500 ppm
I20260924 10:12:54.320216 27128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:54 dist-test-slave-s3n3 krb5kdc[27115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:54.323268 27128 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:12:54.324641 27128 webserver.cc:460] Webserver started at http://127.0.0.1:33135/ using document root <none> and password file <none>
I20260924 10:12:54.325295 27128 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:54.325433 27128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:54.325721 27128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:54.328565 27128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "19079c8c43eb424eb98b512f0dd29477"
format_stamp: "Formatted at 2026-09-24 10:12:54 on dist-test-slave-s3n3"
server_key: "ee401164a1970bff656f5de37ec2125e"
server_key_iv: "309c06541ca965a40543e67395d08ced"
server_key_version: "encryptionkey@0"
I20260924 10:12:54.329288 27128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "19079c8c43eb424eb98b512f0dd29477"
format_stamp: "Formatted at 2026-09-24 10:12:54 on dist-test-slave-s3n3"
server_key: "ee401164a1970bff656f5de37ec2125e"
server_key_iv: "309c06541ca965a40543e67395d08ced"
server_key_version: "encryptionkey@0"
I20260924 10:12:54.334050 27128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:12:54.337386 27143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:54.338599 27128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:12:54.338760 27128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "19079c8c43eb424eb98b512f0dd29477"
format_stamp: "Formatted at 2026-09-24 10:12:54 on dist-test-slave-s3n3"
server_key: "ee401164a1970bff656f5de37ec2125e"
server_key_iv: "309c06541ca965a40543e67395d08ced"
server_key_version: "encryptionkey@0"
I20260924 10:12:54.338945 27128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:54.344254 27128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:54.347931 27128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:54.348212 27128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:54.366384 27128 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35521
I20260924 10:12:54.366456 27194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35521 every 8 connection(s)
I20260924 10:12:54.367763 27128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:54.371436 27195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:54.377089 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 27128
I20260924 10:12:54.377259 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:54.377676 6155 external_mini_cluster.cc:1428] Setting key c46a3b4e8bbd21d54f4577c954e83874
I20260924 10:12:54.382501 27195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477: Bootstrap starting.
I20260924 10:12:54.386307 27198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:54.379505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45573 (local address 127.0.0.1:35521)
0924 10:12:54.380378 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:54.380393 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:54.380439 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:12:54.382681 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:54.382706 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:54.383123 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:54.383518 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:54.384513 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:54.384521 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:54.384539 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:54.384558 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:54.384629 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:54.384656 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:54.384864 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:54.384984 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 10:12:54.385151 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":399,"threads_started":1}
I20260924 10:12:54.386412 27195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:54.387533 27195 log.cc:824] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:54.389909 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:12:54.390535 27195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477: No bootstrap required, opened a new log
I20260924 10:12:54.396368 27198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:54.394012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45273 (local address 127.0.0.1:35521)
0924 10:12:54.394166 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:54.394173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:54.394582 (+ 409us) server_negotiation.cc:407] Connection header received
0924 10:12:54.394710 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:54.394716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:54.394793 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:54.394899 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:54.395773 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:54.395779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:54.395785 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:54.395807 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:54.395848 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:54.395867 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:54.395998 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:54.396040 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 10:12:54.396095 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:12:54.398581 27195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19079c8c43eb424eb98b512f0dd29477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } }
I20260924 10:12:54.398910 27195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:54.398979 27195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19079c8c43eb424eb98b512f0dd29477, State: Initialized, Role: FOLLOWER
I20260924 10:12:54.399509 27195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19079c8c43eb424eb98b512f0dd29477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } }
I20260924 10:12:54.399711 27195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:54.399839 27195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:54.399998 27195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:54.402103 27195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19079c8c43eb424eb98b512f0dd29477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } }
I20260924 10:12:54.402552 27195 leader_election.cc:302] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19079c8c43eb424eb98b512f0dd29477; no voters:
I20260924 10:12:54.403208 27195 leader_election.cc:288] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:54.403321 27205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:54.404006 27205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [term 1 LEADER]: Becoming Leader. State: Replica: 19079c8c43eb424eb98b512f0dd29477, State: Running, Role: LEADER
I20260924 10:12:54.404520 27205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19079c8c43eb424eb98b512f0dd29477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } }
I20260924 10:12:54.405189 27195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:54.407198 27206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19079c8c43eb424eb98b512f0dd29477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19079c8c43eb424eb98b512f0dd29477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } } }
I20260924 10:12:54.407702 27206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:54.408499 27207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19079c8c43eb424eb98b512f0dd29477. Latest consensus state: current_term: 1 leader_uuid: "19079c8c43eb424eb98b512f0dd29477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19079c8c43eb424eb98b512f0dd29477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } } }
I20260924 10:12:54.408896 27207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477 [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:54.412240 27213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:54.417228 27213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:54.425504 27213 catalog_manager.cc:1347] Generated new cluster ID: ab711a961f3f4b3e8325915e80e2c209
I20260924 10:12:54.425611 27213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:54.444828 27213 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:54.445217 27213 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:54.453256 27213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19079c8c43eb424eb98b512f0dd29477: Generated new TSK 0
I20260924 10:12:54.454174 27213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:12:54.461244 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27128
W20260924 10:12:54.462502 27190 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45573
W20260924 10:12:54.463265 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27128: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:12:54.463403 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 27128
2026-09-24T10:12:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1902 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:12:54 dist-test-slave-s3n3 krb5kdc[27226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:12:54 dist-test-slave-s3n3 krb5kdc[27226](info): set up 2 sockets
Sep 24 10:12:54 dist-test-slave-s3n3 krb5kdc[27226](info): commencing operation
krb5kdc: starting...
W20260924 10:12:55.659436 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:12:55 dist-test-slave-s3n3 krb5kdc[27226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:12:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:12:55.904542 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:56.228744 27239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:56.229094 27239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:56.229270 27239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:56.236400 27239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:56.236546 27239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:56.236630 27239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:56.236718 27239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:56.236811 27239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:56.246466 27239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:56.247757 27239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:56.249028 27239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:56.260650 27244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:56.260581 27245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:56.260627 27247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:56.261811 27239 server_base.cc:1034] running on GCE node
I20260924 10:12:56.262543 27239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:56.263962 27239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:56.265164 27239 hybrid_clock.cc:648] HybridClock initialized: now 1790244776265131 us; error 31 us; skew 500 ppm
I20260924 10:12:56.265511 27239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:56 dist-test-slave-s3n3 krb5kdc[27226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:56.274148 27239 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:12:56.275758 27239 webserver.cc:460] Webserver started at http://127.0.0.1:41597/ using document root <none> and password file <none>
I20260924 10:12:56.276451 27239 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:56.276578 27239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:56.276883 27239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:56.279875 27239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "e80b24b0e7be4d838bac9478f121971f"
format_stamp: "Formatted at 2026-09-24 10:12:56 on dist-test-slave-s3n3"
server_key: "d25c087191f4b053ae1a5c195a759057"
server_key_iv: "3967a94d9f26efc8b15ff7535ea3eca4"
server_key_version: "encryptionkey@0"
I20260924 10:12:56.280629 27239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "e80b24b0e7be4d838bac9478f121971f"
format_stamp: "Formatted at 2026-09-24 10:12:56 on dist-test-slave-s3n3"
server_key: "d25c087191f4b053ae1a5c195a759057"
server_key_iv: "3967a94d9f26efc8b15ff7535ea3eca4"
server_key_version: "encryptionkey@0"
I20260924 10:12:56.285583 27239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:12:56.289052 27254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:56.290336 27239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:12:56.290537 27239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "e80b24b0e7be4d838bac9478f121971f"
format_stamp: "Formatted at 2026-09-24 10:12:56 on dist-test-slave-s3n3"
server_key: "d25c087191f4b053ae1a5c195a759057"
server_key_iv: "3967a94d9f26efc8b15ff7535ea3eca4"
server_key_version: "encryptionkey@0"
I20260924 10:12:56.290737 27239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:56.313388 27239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:56.317479 27239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:56.317772 27239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:56.335947 27239 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40315
I20260924 10:12:56.335999 27305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40315 every 8 connection(s)
I20260924 10:12:56.337265 27239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:56.340811 27306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:56.347415 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 27239
I20260924 10:12:56.347592 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:56.348148 6155 external_mini_cluster.cc:1428] Setting key f876225bbbde9a7984307633705fba7d
I20260924 10:12:56.351449 27306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f: Bootstrap starting.
I20260924 10:12:56.354786 27306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:56.355983 27306 log.cc:824] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:56.359157 27306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f: No bootstrap required, opened a new log
Sep 24 10:12:56 dist-test-slave-s3n3 krb5kdc[27226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:12:56.367833 27306 raft_consensus.cc:348] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80b24b0e7be4d838bac9478f121971f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } }
I20260924 10:12:56.368203 27306 raft_consensus.cc:374] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:56.368299 27306 raft_consensus.cc:729] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e80b24b0e7be4d838bac9478f121971f, State: Initialized, Role: FOLLOWER
I20260924 10:12:56.368950 27306 consensus_queue.cc:260] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80b24b0e7be4d838bac9478f121971f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } }
I20260924 10:12:56.369184 27306 raft_consensus.cc:388] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:56.369319 27306 raft_consensus.cc:482] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:56.369515 27306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:56.372255 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:56.350054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:45541 (local address 127.0.0.1:40315)
0924 10:12:56.350849 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:56.350860 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:56.350899 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:12:56.352673 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:56.352694 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:56.353075 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:56.353385 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:56.355177 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:56.356784 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:56.358562 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:56.359372 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:56.359507 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:56.364528 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:56.364537 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:56.364552 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:56.364591 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:56.367709 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:56.368614 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:56.368620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:56.368626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:56.368788 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:56.369253 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:56.369263 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:56.369265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:56.369903 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:56.370266 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:56.370768 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 10:12:56.371313 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":345,"threads_started":1}
I20260924 10:12:56.372295 27306 raft_consensus.cc:504] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80b24b0e7be4d838bac9478f121971f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } }
I20260924 10:12:56.372869 27306 leader_election.cc:302] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e80b24b0e7be4d838bac9478f121971f; no voters:
I20260924 10:12:56.373493 27306 leader_election.cc:288] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:56.373632 27311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:56.374779 27311 raft_consensus.cc:686] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [term 1 LEADER]: Becoming Leader. State: Replica: e80b24b0e7be4d838bac9478f121971f, State: Running, Role: LEADER
I20260924 10:12:56.375417 27311 consensus_queue.cc:237] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80b24b0e7be4d838bac9478f121971f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } }
I20260924 10:12:56.375775 27306 sys_catalog.cc:564] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:56.376550 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:12:56.378150 27313 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e80b24b0e7be4d838bac9478f121971f. Latest consensus state: current_term: 1 leader_uuid: "e80b24b0e7be4d838bac9478f121971f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80b24b0e7be4d838bac9478f121971f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } } }
I20260924 10:12:56.378576 27312 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e80b24b0e7be4d838bac9478f121971f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80b24b0e7be4d838bac9478f121971f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } } }
I20260924 10:12:56.380429 27313 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:56.380678 27312 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:56.385922 27324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:56.390972 27324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:56.400962 27324 catalog_manager.cc:1347] Generated new cluster ID: 58d67dae0eaa4b4cb87e7e784e973188
I20260924 10:12:56.401067 27324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:56.402316 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:56.382854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:56073 (local address 127.0.0.1:40315)
0924 10:12:56.383042 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:56.383048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:56.384473 (+ 1425us) server_negotiation.cc:407] Connection header received
0924 10:12:56.384696 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:56.384705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:56.384821 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:56.384953 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:56.387477 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:56.388290 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:56.390567 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:56.391279 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:56.391435 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:56.395266 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:56.395273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:56.395279 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:12:56.395324 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:56.398741 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:56.399756 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:56.399761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:56.399765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:56.399924 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:12:56.400530 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:12:56.400535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:12:56.400538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:12:56.400976 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:56.401197 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:56.401765 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 10:12:56.401952 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:12:56.421065 27324 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:56.422312 27324 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:56.430617 27324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e80b24b0e7be4d838bac9478f121971f: Generated new TSK 0
I20260924 10:12:56.431478 27324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:12:56.437764 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27239
I20260924 10:12:56.438889 27301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:12:56.527874 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 27239
W20260924 10:12:56.544983 27224 connection.cc:535] client connection to 127.0.0.1:40315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40315): Connection reset by peer (error 104)
2026-09-24T10:12:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2067 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:12:56 dist-test-slave-s3n3 krb5kdc[27338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:12:56 dist-test-slave-s3n3 krb5kdc[27338](info): set up 2 sockets
Sep 24 10:12:56 dist-test-slave-s3n3 krb5kdc[27338](info): commencing operation
krb5kdc: starting...
W20260924 10:12:57.631513 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:12:57 dist-test-slave-s3n3 krb5kdc[27338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:12:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:12:57.899506 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45205
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:12:58.234179 27351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:12:58.234475 27351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:12:58.234652 27351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:12:58.242084 27351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:12:58.242204 27351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:12:58.242301 27351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:12:58.242408 27351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:12:58.242468 27351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:12:58.252609 27351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:12:58.253976 27351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:12:58.255314 27351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:12:58.267162 27356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:58.267230 27359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:12:58.267108 27357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:12:58.268198 27351 server_base.cc:1034] running on GCE node
I20260924 10:12:58.269052 27351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:12:58.270460 27351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:12:58.271663 27351 hybrid_clock.cc:648] HybridClock initialized: now 1790244778271602 us; error 49 us; skew 500 ppm
I20260924 10:12:58.272104 27351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:12:58 dist-test-slave-s3n3 krb5kdc[27338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:12:58.275381 27351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:12:58.276794 27351 webserver.cc:460] Webserver started at http://127.0.0.1:36153/ using document root <none> and password file <none>
I20260924 10:12:58.277486 27351 fs_manager.cc:359] Metadata directory not provided
I20260924 10:12:58.277616 27351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:12:58.277940 27351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:12:58.280952 27351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "96242634dc664351bfe6b70f5ac9a5ab"
format_stamp: "Formatted at 2026-09-24 10:12:58 on dist-test-slave-s3n3"
server_key: "0518ca8acf91566b6a0bd05b3b888687"
server_key_iv: "416cb46bacf7bcc9991a9d10a4ada957"
server_key_version: "encryptionkey@0"
I20260924 10:12:58.281795 27351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "96242634dc664351bfe6b70f5ac9a5ab"
format_stamp: "Formatted at 2026-09-24 10:12:58 on dist-test-slave-s3n3"
server_key: "0518ca8acf91566b6a0bd05b3b888687"
server_key_iv: "416cb46bacf7bcc9991a9d10a4ada957"
server_key_version: "encryptionkey@0"
I20260924 10:12:58.286960 27351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 10:12:58.290416 27366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:12:58.291858 27351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:12:58.292065 27351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "96242634dc664351bfe6b70f5ac9a5ab"
format_stamp: "Formatted at 2026-09-24 10:12:58 on dist-test-slave-s3n3"
server_key: "0518ca8acf91566b6a0bd05b3b888687"
server_key_iv: "416cb46bacf7bcc9991a9d10a4ada957"
server_key_version: "encryptionkey@0"
I20260924 10:12:58.292268 27351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:12:58.305804 27351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:12:58.310045 27351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:12:58.310340 27351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:12:58.329164 27351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45205
I20260924 10:12:58.329205 27417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45205 every 8 connection(s)
I20260924 10:12:58.330576 27351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:12:58.334298 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 27351
I20260924 10:12:58.334484 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:12:58.334550 27418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:12:58.334954 6155 external_mini_cluster.cc:1428] Setting key 2f32e0a0e5bb7c414021fa7111a2acad
I20260924 10:12:58.349944 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:58.336844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:33411 (local address 127.0.0.1:45205)
0924 10:12:58.337683 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:58.337697 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:58.337763 (+ 66us) server_negotiation.cc:407] Connection header received
0924 10:12:58.339733 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:58.339762 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:58.340328 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:58.340782 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:58.341321 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:58.343372 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:58.344713 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:58.345742 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:58.345945 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:58.347163 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:58.347173 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:58.347191 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:58.347947 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:58.348051 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:58.348083 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:58.348266 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:58.348478 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 10:12:58.348645 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":318,"threads_started":1}
I20260924 10:12:58.350469 27418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab: Bootstrap starting.
I20260924 10:12:58.353657 27418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab: Neither blocks nor log segments found. Creating new log.
I20260924 10:12:58.353710 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:12:58.354883 27418 log.cc:824] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab: Log is configured to *not* fsync() on all Append() calls
I20260924 10:12:58.358341 27418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab: No bootstrap required, opened a new log
I20260924 10:12:58.365890 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:12:58.358963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:44263 (local address 127.0.0.1:45205)
0924 10:12:58.359160 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 10:12:58.359166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:12:58.359515 (+ 349us) server_negotiation.cc:407] Connection header received
0924 10:12:58.359663 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:12:58.359668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:12:58.359755 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:12:58.359877 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:12:58.360375 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:58.361690 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:58.363119 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:12:58.363897 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:12:58.364033 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:12:58.365099 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:12:58.365110 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:12:58.365115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:12:58.365137 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:12:58.365187 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:12:58.365208 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:12:58.365325 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 10:12:58.365471 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 10:12:58.365528 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:12:58.368314 27418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96242634dc664351bfe6b70f5ac9a5ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45205 } }
I20260924 10:12:58.368696 27418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:12:58.368821 27418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96242634dc664351bfe6b70f5ac9a5ab, State: Initialized, Role: FOLLOWER
I20260924 10:12:58.369412 27418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96242634dc664351bfe6b70f5ac9a5ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45205 } }
I20260924 10:12:58.369657 27418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:12:58.369803 27418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:12:58.369984 27418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:12:58.372143 27418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96242634dc664351bfe6b70f5ac9a5ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45205 } }
I20260924 10:12:58.372640 27418 leader_election.cc:302] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96242634dc664351bfe6b70f5ac9a5ab; no voters:
I20260924 10:12:58.373191 27418 leader_election.cc:288] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:12:58.373335 27428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:12:58.374028 27428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [term 1 LEADER]: Becoming Leader. State: Replica: 96242634dc664351bfe6b70f5ac9a5ab, State: Running, Role: LEADER
I20260924 10:12:58.374568 27428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96242634dc664351bfe6b70f5ac9a5ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45205 } }
I20260924 10:12:58.375349 27418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:12:58.377660 27429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96242634dc664351bfe6b70f5ac9a5ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96242634dc664351bfe6b70f5ac9a5ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45205 } } }
I20260924 10:12:58.377669 27430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96242634dc664351bfe6b70f5ac9a5ab. Latest consensus state: current_term: 1 leader_uuid: "96242634dc664351bfe6b70f5ac9a5ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96242634dc664351bfe6b70f5ac9a5ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45205 } } }
I20260924 10:12:58.378103 27429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:58.378103 27430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab [sys.catalog]: This master's current role is: LEADER
I20260924 10:12:58.379030 27434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:12:58.383885 27434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:12:58.392700 27434 catalog_manager.cc:1347] Generated new cluster ID: 58ecaf85599d47d0a8adaea45c6fce37
I20260924 10:12:58.392886 27434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:12:58.402848 27434 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:12:58.404093 27434 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:12:58.411413 27434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96242634dc664351bfe6b70f5ac9a5ab: Generated new TSK 0
I20260924 10:12:58.412204 27434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:12:58.425593 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27351
W20260924 10:12:58.426774 27413 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.211:33411
W20260924 10:12:58.427577 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27351: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:12:58.427717 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 27351
W20260924 10:12:58.444674 27336 connection.cc:535] client connection to 127.0.0.1:45205 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45205): Connection reset by peer (error 104)
2026-09-24T10:12:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1899 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:12:58 dist-test-slave-s3n3 krb5kdc[27449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:12:58 dist-test-slave-s3n3 krb5kdc[27449](info): set up 2 sockets
Sep 24 10:12:58 dist-test-slave-s3n3 krb5kdc[27449](info): commencing operation
krb5kdc: starting...
W20260924 10:12:59.539561 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:12:59 dist-test-slave-s3n3 krb5kdc[27449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:12:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:12:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:12:59.795365 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:13:00.134189 27462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:00.134531 27462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:00.134688 27462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:00.142077 27462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:00.142226 27462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:00.142328 27462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:00.142431 27462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:00.142503 27462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:00.152633 27462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:00.153951 27462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:00.155265 27462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:00.167150 27467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:00.167034 27470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:00.167124 27468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:00.168179 27462 server_base.cc:1034] running on GCE node
I20260924 10:13:00.169184 27462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:00.171172 27462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:00.172421 27462 hybrid_clock.cc:648] HybridClock initialized: now 1790244780172367 us; error 34 us; skew 500 ppm
I20260924 10:13:00.172734 27462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:00 dist-test-slave-s3n3 krb5kdc[27449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:00.176705 27462 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:13:00.178205 27462 webserver.cc:460] Webserver started at http://127.0.0.1:36015/ using document root <none> and password file <none>
I20260924 10:13:00.178879 27462 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:00.179004 27462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:00.179333 27462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:00.182343 27462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "a33a8b1fe6334ad4a3ad308691aa96dc"
format_stamp: "Formatted at 2026-09-24 10:13:00 on dist-test-slave-s3n3"
server_key: "e0ce030a211f6cb60ceb9cf5d631e0cc"
server_key_iv: "55dfc78cc5c45fbebd958aac959f33bc"
server_key_version: "encryptionkey@0"
I20260924 10:13:00.183450 27462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "a33a8b1fe6334ad4a3ad308691aa96dc"
format_stamp: "Formatted at 2026-09-24 10:13:00 on dist-test-slave-s3n3"
server_key: "e0ce030a211f6cb60ceb9cf5d631e0cc"
server_key_iv: "55dfc78cc5c45fbebd958aac959f33bc"
server_key_version: "encryptionkey@0"
I20260924 10:13:00.189208 27462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:13:00.192680 27477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:00.193935 27462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 10:13:00.194108 27462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "a33a8b1fe6334ad4a3ad308691aa96dc"
format_stamp: "Formatted at 2026-09-24 10:13:00 on dist-test-slave-s3n3"
server_key: "e0ce030a211f6cb60ceb9cf5d631e0cc"
server_key_iv: "55dfc78cc5c45fbebd958aac959f33bc"
server_key_version: "encryptionkey@0"
I20260924 10:13:00.194303 27462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:00.200064 27462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:00.204285 27462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:00.204578 27462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:00.222829 27462 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40413
I20260924 10:13:00.222882 27528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40413 every 8 connection(s)
I20260924 10:13:00.224259 27462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:13:00.227871 27529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:00.232024 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 27462
I20260924 10:13:00.232194 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:13:00.232720 6155 external_mini_cluster.cc:1428] Setting key cae429200b35469c26c1b6dffc1bcae6
I20260924 10:13:00.240181 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:00.234516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:47269 (local address 127.0.0.1:40413)
0924 10:13:00.235313 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:00.235324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:00.235363 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:13:00.236946 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:00.236967 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:00.237296 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:00.237607 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:00.238618 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:00.238627 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:00.238639 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:00.238653 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:00.238713 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:00.238732 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:00.238970 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:00.239140 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 10:13:00.239331 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":355,"threads_started":1}
I20260924 10:13:00.240909 27529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc: Bootstrap starting.
I20260924 10:13:00.243943 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:13:00.244680 27529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:00.245904 27529 log.cc:824] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc: Log is configured to *not* fsync() on all Append() calls
I20260924 10:13:00.248618 27529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc: No bootstrap required, opened a new log
I20260924 10:13:00.251672 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:00.249127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:46557 (local address 127.0.0.1:40413)
0924 10:13:00.249354 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:00.249364 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:00.249764 (+ 400us) server_negotiation.cc:407] Connection header received
0924 10:13:00.249921 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:00.249925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:00.249977 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:00.250059 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:00.250977 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:00.250982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:00.250986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:00.251000 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:00.251038 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:00.251054 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:00.251200 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:00.251307 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 10:13:00.251354 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:13:00.256911 27529 raft_consensus.cc:348] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33a8b1fe6334ad4a3ad308691aa96dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40413 } }
I20260924 10:13:00.257297 27529 raft_consensus.cc:374] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:13:00.257402 27529 raft_consensus.cc:729] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33a8b1fe6334ad4a3ad308691aa96dc, State: Initialized, Role: FOLLOWER
I20260924 10:13:00.258081 27529 consensus_queue.cc:260] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33a8b1fe6334ad4a3ad308691aa96dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40413 } }
I20260924 10:13:00.258352 27529 raft_consensus.cc:388] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:13:00.258502 27529 raft_consensus.cc:482] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:13:00.258677 27529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:00.260867 27529 raft_consensus.cc:504] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33a8b1fe6334ad4a3ad308691aa96dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40413 } }
I20260924 10:13:00.261334 27529 leader_election.cc:302] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a33a8b1fe6334ad4a3ad308691aa96dc; no voters:
I20260924 10:13:00.261893 27529 leader_election.cc:288] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:13:00.262053 27539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:13:00.262714 27539 raft_consensus.cc:686] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [term 1 LEADER]: Becoming Leader. State: Replica: a33a8b1fe6334ad4a3ad308691aa96dc, State: Running, Role: LEADER
I20260924 10:13:00.263389 27539 consensus_queue.cc:237] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33a8b1fe6334ad4a3ad308691aa96dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40413 } }
I20260924 10:13:00.263969 27529 sys_catalog.cc:564] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:00.266271 27541 sys_catalog.cc:455] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a33a8b1fe6334ad4a3ad308691aa96dc. Latest consensus state: current_term: 1 leader_uuid: "a33a8b1fe6334ad4a3ad308691aa96dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33a8b1fe6334ad4a3ad308691aa96dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40413 } } }
I20260924 10:13:00.266613 27541 sys_catalog.cc:458] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:00.267549 27546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:13:00.267477 27540 sys_catalog.cc:455] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a33a8b1fe6334ad4a3ad308691aa96dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33a8b1fe6334ad4a3ad308691aa96dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40413 } } }
I20260924 10:13:00.267798 27540 sys_catalog.cc:458] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:00.273036 27546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:13:00.281719 27546 catalog_manager.cc:1347] Generated new cluster ID: 734db2546f704b4b9cfbf7640646eea1
I20260924 10:13:00.281838 27546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:13:00.300843 27546 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:13:00.301259 27546 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:13:00.314848 27546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a33a8b1fe6334ad4a3ad308691aa96dc: Generated new TSK 0
I20260924 10:13:00.315719 27546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:13:00.320171 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27462
W20260924 10:13:00.321532 27524 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.211:47269
W20260924 10:13:00.322222 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27462: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:13:00.322376 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 27462
2026-09-24T10:13:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1894 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:13:00 dist-test-slave-s3n3 krb5kdc[27560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:13:00 dist-test-slave-s3n3 krb5kdc[27560](info): set up 2 sockets
Sep 24 10:13:00 dist-test-slave-s3n3 krb5kdc[27560](info): commencing operation
krb5kdc: starting...
W20260924 10:13:01.435576 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:13:01 dist-test-slave-s3n3 krb5kdc[27560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:13:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:13:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:13:01.694710 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:13:02.018857 27573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:02.019243 27573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:02.019416 27573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:02.026785 27573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:02.026942 27573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:02.027050 27573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:02.027185 27573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:02.027264 27573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:02.037508 27573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:02.038823 27573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:02.040163 27573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:02.051936 27579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:02.052031 27578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:02.051959 27581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:02.052800 27573 server_base.cc:1034] running on GCE node
I20260924 10:13:02.053577 27573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:02.054950 27573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:02.056173 27573 hybrid_clock.cc:648] HybridClock initialized: now 1790244782056146 us; error 70 us; skew 500 ppm
I20260924 10:13:02.056520 27573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:02 dist-test-slave-s3n3 krb5kdc[27560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:02.066154 27573 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:13:02.067816 27573 webserver.cc:460] Webserver started at http://127.0.0.1:42957/ using document root <none> and password file <none>
I20260924 10:13:02.068534 27573 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:02.068678 27573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:02.068982 27573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:02.072062 27573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "479273d0fc134019bd7b971ca44d08d1"
format_stamp: "Formatted at 2026-09-24 10:13:02 on dist-test-slave-s3n3"
server_key: "6683f86ec77e39356562680232ac3af3"
server_key_iv: "93b118e98e6da7edb061b64fd811129f"
server_key_version: "encryptionkey@0"
I20260924 10:13:02.072889 27573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "479273d0fc134019bd7b971ca44d08d1"
format_stamp: "Formatted at 2026-09-24 10:13:02 on dist-test-slave-s3n3"
server_key: "6683f86ec77e39356562680232ac3af3"
server_key_iv: "93b118e98e6da7edb061b64fd811129f"
server_key_version: "encryptionkey@0"
I20260924 10:13:02.078174 27573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:13:02.081631 27588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:02.082877 27573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:13:02.083052 27573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "479273d0fc134019bd7b971ca44d08d1"
format_stamp: "Formatted at 2026-09-24 10:13:02 on dist-test-slave-s3n3"
server_key: "6683f86ec77e39356562680232ac3af3"
server_key_iv: "93b118e98e6da7edb061b64fd811129f"
server_key_version: "encryptionkey@0"
I20260924 10:13:02.083308 27573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:02.100786 27573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:02.104800 27573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:02.105094 27573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:02.123845 27573 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35905
I20260924 10:13:02.123878 27639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35905 every 8 connection(s)
I20260924 10:13:02.125276 27573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:13:02.129041 27640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:02.129182 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 27573
I20260924 10:13:02.129356 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:13:02.129917 6155 external_mini_cluster.cc:1428] Setting key 4ca9d244ed54131f4f484228188610d9
I20260924 10:13:02.141760 27640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1: Bootstrap starting.
Sep 24 10:13:02 dist-test-slave-s3n3 krb5kdc[27560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:13:02.145318 27640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:02.146548 27640 log.cc:824] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:13:02.149930 27640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1: No bootstrap required, opened a new log
I20260924 10:13:02.152690 27643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:02.131707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:49325 (local address 127.0.0.1:35905)
0924 10:13:02.132501 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:02.132512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:02.132553 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:13:02.134207 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:02.134227 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:02.134596 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:02.134983 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:02.136458 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:02.137709 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:02.139084 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:02.140015 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:02.140231 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:02.144802 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:02.144810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:02.144824 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:02.144864 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:02.148340 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:02.149251 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:02.149257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:02.149263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:02.149417 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:02.149899 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:02.149903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:02.149904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:02.150514 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:02.150765 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:02.151350 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 10:13:02.151662 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":357,"threads_started":1}
I20260924 10:13:02.156500 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:13:02.158823 27640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479273d0fc134019bd7b971ca44d08d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260924 10:13:02.159307 27640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:13:02.159422 27640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479273d0fc134019bd7b971ca44d08d1, State: Initialized, Role: FOLLOWER
I20260924 10:13:02.160112 27640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479273d0fc134019bd7b971ca44d08d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260924 10:13:02.160446 27640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:13:02.160617 27640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:13:02.160775 27640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:02.163609 27640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479273d0fc134019bd7b971ca44d08d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260924 10:13:02.164091 27640 leader_election.cc:302] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 479273d0fc134019bd7b971ca44d08d1; no voters:
I20260924 10:13:02.164762 27640 leader_election.cc:288] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:13:02.164891 27650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:13:02.165551 27650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [term 1 LEADER]: Becoming Leader. State: Replica: 479273d0fc134019bd7b971ca44d08d1, State: Running, Role: LEADER
I20260924 10:13:02.166121 27650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479273d0fc134019bd7b971ca44d08d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260924 10:13:02.168700 27652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "479273d0fc134019bd7b971ca44d08d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479273d0fc134019bd7b971ca44d08d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } } }
I20260924 10:13:02.169123 27652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:02.172443 27650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 479273d0fc134019bd7b971ca44d08d1. Latest consensus state: current_term: 1 leader_uuid: "479273d0fc134019bd7b971ca44d08d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479273d0fc134019bd7b971ca44d08d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } } }
I20260924 10:13:02.172819 27650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:02.175500 27640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:02.175820 27654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:13:02.182415 27654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:13:02.182587 27643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:02.161826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:43211 (local address 127.0.0.1:35905)
0924 10:13:02.162014 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:02.162020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:02.162397 (+ 377us) server_negotiation.cc:407] Connection header received
0924 10:13:02.162542 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:02.162548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:02.162628 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:02.162742 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:02.164401 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:02.165937 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:02.168468 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:02.169606 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:02.169805 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:02.173523 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:02.173530 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:02.173534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:02.173570 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:02.178257 (+ 4687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:02.179362 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:02.179368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:02.179372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:02.179519 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:02.180113 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:02.180119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:02.180124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:02.180543 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:02.180858 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:02.181970 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0924 10:13:02.182168 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 10:13:02.194221 27654 catalog_manager.cc:1347] Generated new cluster ID: 8172a14c7b4b496eb8dccb89c79248e6
I20260924 10:13:02.194736 27654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:13:02.205998 27654 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:13:02.207825 27654 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:13:02.220247 27654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 479273d0fc134019bd7b971ca44d08d1: Generated new TSK 0
I20260924 10:13:02.221302 27654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:13:02.235020 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27573
I20260924 10:13:02.236707 27635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:13:02.327654 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 27573
W20260924 10:13:02.345103 27558 connection.cc:535] client connection to 127.0.0.1:35905 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35905): Connection reset by peer (error 104)
2026-09-24T10:13:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2005 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:13:02 dist-test-slave-s3n3 krb5kdc[27672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:13:02 dist-test-slave-s3n3 krb5kdc[27672](info): set up 2 sockets
Sep 24 10:13:02 dist-test-slave-s3n3 krb5kdc[27672](info): commencing operation
krb5kdc: starting...
W20260924 10:13:03.435534 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:13:03 dist-test-slave-s3n3 krb5kdc[27672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:13:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:13:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:13:03.706139 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:13:04.032929 27685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:04.033212 27685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:04.033385 27685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:04.040657 27685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:04.040764 27685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:04.040880 27685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:04.040980 27685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:04.041054 27685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:04.050981 27685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40133
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:04.052338 27685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:04.053617 27685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:04.065213 27691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:04.065292 27690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:04.065258 27693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:04.066111 27685 server_base.cc:1034] running on GCE node
I20260924 10:13:04.066856 27685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:04.068331 27685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:04.069566 27685 hybrid_clock.cc:648] HybridClock initialized: now 1790244784069527 us; error 51 us; skew 500 ppm
I20260924 10:13:04.069929 27685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:04 dist-test-slave-s3n3 krb5kdc[27672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:04.073066 27685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:13:04.074476 27685 webserver.cc:460] Webserver started at http://127.0.0.1:33461/ using document root <none> and password file <none>
I20260924 10:13:04.075242 27685 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:04.075387 27685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:04.075690 27685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:04.078599 27685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "df02f823a6d1409b901b8c02db9ec4cb"
format_stamp: "Formatted at 2026-09-24 10:13:04 on dist-test-slave-s3n3"
server_key: "f348a3e96467d942c6ed0a6fcd8b6db6"
server_key_iv: "1e56761453ff30bee3828a1de2abf9d8"
server_key_version: "encryptionkey@0"
I20260924 10:13:04.079434 27685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "df02f823a6d1409b901b8c02db9ec4cb"
format_stamp: "Formatted at 2026-09-24 10:13:04 on dist-test-slave-s3n3"
server_key: "f348a3e96467d942c6ed0a6fcd8b6db6"
server_key_iv: "1e56761453ff30bee3828a1de2abf9d8"
server_key_version: "encryptionkey@0"
I20260924 10:13:04.084602 27685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:13:04.088016 27700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:04.089278 27685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:13:04.089463 27685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "df02f823a6d1409b901b8c02db9ec4cb"
format_stamp: "Formatted at 2026-09-24 10:13:04 on dist-test-slave-s3n3"
server_key: "f348a3e96467d942c6ed0a6fcd8b6db6"
server_key_iv: "1e56761453ff30bee3828a1de2abf9d8"
server_key_version: "encryptionkey@0"
I20260924 10:13:04.089685 27685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:04.124063 27685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:04.128252 27685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:04.128571 27685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:04.147341 27685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40133
I20260924 10:13:04.147372 27751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40133 every 8 connection(s)
I20260924 10:13:04.148744 27685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:13:04.150888 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 27685
I20260924 10:13:04.151103 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:13:04.151644 6155 external_mini_cluster.cc:1428] Setting key d96289c34e4df368ecc72045e7a1479c
I20260924 10:13:04.153071 27752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:04.165000 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:04.153485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:34519 (local address 127.0.0.1:40133)
0924 10:13:04.154258 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:04.154270 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:04.154310 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:13:04.155970 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:04.155991 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:04.156389 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:04.156726 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:04.157304 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:04.158642 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:04.160550 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:04.161203 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:04.161333 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:04.162642 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:04.162649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:04.162663 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:04.163306 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:04.163383 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:04.163410 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:04.163540 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:04.163827 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 10:13:04.164019 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":342,"threads_started":1}
I20260924 10:13:04.166188 27752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb: Bootstrap starting.
I20260924 10:13:04.169091 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:13:04.169988 27752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:04.171356 27752 log.cc:824] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb: Log is configured to *not* fsync() on all Append() calls
I20260924 10:13:04.175913 27752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb: No bootstrap required, opened a new log
I20260924 10:13:04.183864 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:04.176902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:52961 (local address 127.0.0.1:40133)
0924 10:13:04.177068 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:04.177074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:04.177834 (+ 760us) server_negotiation.cc:407] Connection header received
0924 10:13:04.178255 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:04.178261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:04.178348 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:04.178480 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:04.179121 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:04.179939 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:04.181252 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:04.181748 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:04.181889 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:04.183096 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:04.183103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:04.183108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:04.183129 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:04.183174 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:04.183193 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:04.183332 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:04.183524 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 10:13:04.183585 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:13:04.184499 27752 raft_consensus.cc:348] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df02f823a6d1409b901b8c02db9ec4cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40133 } }
I20260924 10:13:04.184883 27752 raft_consensus.cc:374] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:13:04.185003 27752 raft_consensus.cc:729] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df02f823a6d1409b901b8c02db9ec4cb, State: Initialized, Role: FOLLOWER
I20260924 10:13:04.185606 27752 consensus_queue.cc:260] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df02f823a6d1409b901b8c02db9ec4cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40133 } }
I20260924 10:13:04.185829 27752 raft_consensus.cc:388] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:13:04.185986 27752 raft_consensus.cc:482] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:13:04.186169 27752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:04.188510 27752 raft_consensus.cc:504] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df02f823a6d1409b901b8c02db9ec4cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40133 } }
I20260924 10:13:04.189105 27752 leader_election.cc:302] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df02f823a6d1409b901b8c02db9ec4cb; no voters:
I20260924 10:13:04.189742 27752 leader_election.cc:288] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:13:04.189903 27762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:13:04.190603 27762 raft_consensus.cc:686] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [term 1 LEADER]: Becoming Leader. State: Replica: df02f823a6d1409b901b8c02db9ec4cb, State: Running, Role: LEADER
I20260924 10:13:04.191191 27762 consensus_queue.cc:237] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df02f823a6d1409b901b8c02db9ec4cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40133 } }
I20260924 10:13:04.191844 27752 sys_catalog.cc:564] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:04.194614 27763 sys_catalog.cc:455] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df02f823a6d1409b901b8c02db9ec4cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df02f823a6d1409b901b8c02db9ec4cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40133 } } }
I20260924 10:13:04.194638 27764 sys_catalog.cc:455] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader df02f823a6d1409b901b8c02db9ec4cb. Latest consensus state: current_term: 1 leader_uuid: "df02f823a6d1409b901b8c02db9ec4cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df02f823a6d1409b901b8c02db9ec4cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40133 } } }
I20260924 10:13:04.195116 27763 sys_catalog.cc:458] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:04.195116 27764 sys_catalog.cc:458] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb [sys.catalog]: This master's current role is: LEADER
W20260924 10:13:04.205054 27778 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:13:04.205184 27778 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:13:04.205343 27771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:13:04.206914 27771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:13:04.215317 27771 catalog_manager.cc:1347] Generated new cluster ID: e9beb062b77f40a5b9151547c25d81df
I20260924 10:13:04.215418 27771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:13:04.249523 27771 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:13:04.250844 27771 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:13:04.264518 27771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df02f823a6d1409b901b8c02db9ec4cb: Generated new TSK 0
I20260924 10:13:04.265374 27771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:13:04.273823 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27685
W20260924 10:13:04.275489 27747 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.211:34519
W20260924 10:13:04.276264 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27685: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:13:04.276422 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 27685
W20260924 10:13:04.294052 27670 connection.cc:535] client connection to 127.0.0.1:40133 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40133): Connection reset by peer (error 104)
2026-09-24T10:13:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1948 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:13:04 dist-test-slave-s3n3 krb5kdc[27783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:13:04 dist-test-slave-s3n3 krb5kdc[27783](info): set up 2 sockets
Sep 24 10:13:04 dist-test-slave-s3n3 krb5kdc[27783](info): commencing operation
krb5kdc: starting...
W20260924 10:13:05.375545 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:13:05 dist-test-slave-s3n3 krb5kdc[27783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:13:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:13:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:13:05.630896 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:13:05.956029 27796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:05.956434 27796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:05.956612 27796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:05.963788 27796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:05.963936 27796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:05.964051 27796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:05.964160 27796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:05.964247 27796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:05.974027 27796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:05.975376 27796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:05.976649 27796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:05.988554 27802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:05.988523 27801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:05.988701 27804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:05.989627 27796 server_base.cc:1034] running on GCE node
I20260924 10:13:05.990358 27796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:05.991780 27796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:05.993043 27796 hybrid_clock.cc:648] HybridClock initialized: now 1790244785992994 us; error 44 us; skew 500 ppm
I20260924 10:13:05.993387 27796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:05 dist-test-slave-s3n3 krb5kdc[27783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:05.996490 27796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:13:05.997907 27796 webserver.cc:460] Webserver started at http://127.0.0.1:45111/ using document root <none> and password file <none>
I20260924 10:13:05.998595 27796 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:05.998741 27796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:05.999047 27796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:06.002066 27796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "96381f1a35424b8d8ffbcb3e73610fc9"
format_stamp: "Formatted at 2026-09-24 10:13:05 on dist-test-slave-s3n3"
server_key: "8eca388a86c5ae82fe67103e25720c2e"
server_key_iv: "d223bb39e3d8f572230fa0aef024d4f3"
server_key_version: "encryptionkey@0"
I20260924 10:13:06.002820 27796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "96381f1a35424b8d8ffbcb3e73610fc9"
format_stamp: "Formatted at 2026-09-24 10:13:05 on dist-test-slave-s3n3"
server_key: "8eca388a86c5ae82fe67103e25720c2e"
server_key_iv: "d223bb39e3d8f572230fa0aef024d4f3"
server_key_version: "encryptionkey@0"
I20260924 10:13:06.008077 27796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 10:13:06.011534 27811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:06.012876 27796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:13:06.013067 27796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "96381f1a35424b8d8ffbcb3e73610fc9"
format_stamp: "Formatted at 2026-09-24 10:13:05 on dist-test-slave-s3n3"
server_key: "8eca388a86c5ae82fe67103e25720c2e"
server_key_iv: "d223bb39e3d8f572230fa0aef024d4f3"
server_key_version: "encryptionkey@0"
I20260924 10:13:06.013269 27796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:06.018857 27796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:06.022882 27796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:06.023227 27796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:06.041738 27796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34765
I20260924 10:13:06.041778 27862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34765 every 8 connection(s)
I20260924 10:13:06.043182 27796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:13:06.044363 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 27796
I20260924 10:13:06.044541 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:13:06.045096 6155 external_mini_cluster.cc:1428] Setting key a4e012a0acef84a8d44d3a140f582604
I20260924 10:13:06.047803 27863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:06.052596 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:06.046867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:38611 (local address 127.0.0.1:34765)
0924 10:13:06.047613 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:06.047624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:06.047662 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:13:06.049282 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:06.049302 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:06.049597 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:06.049997 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:06.051037 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:06.051043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:06.051054 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:06.051111 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:06.051199 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:06.051220 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:06.051448 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:06.051581 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 10:13:06.051759 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":335,"threads_started":1}
I20260924 10:13:06.056042 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:13:06.058732 27863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9: Bootstrap starting.
I20260924 10:13:06.062155 27863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:06.062920 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:06.060395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:55235 (local address 127.0.0.1:34765)
0924 10:13:06.060597 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:06.060605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:06.060931 (+ 326us) server_negotiation.cc:407] Connection header received
0924 10:13:06.061126 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:06.061131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:06.061184 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:06.061285 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:06.062209 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:06.062214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:06.062219 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:06.062241 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:06.062299 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:06.062321 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:06.062464 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:06.062618 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 10:13:06.062676 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:13:06.063421 27863 log.cc:824] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9: Log is configured to *not* fsync() on all Append() calls
I20260924 10:13:06.066009 27863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9: No bootstrap required, opened a new log
I20260924 10:13:06.073108 27863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96381f1a35424b8d8ffbcb3e73610fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34765 } }
I20260924 10:13:06.073496 27863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:13:06.073601 27863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96381f1a35424b8d8ffbcb3e73610fc9, State: Initialized, Role: FOLLOWER
I20260924 10:13:06.074123 27863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96381f1a35424b8d8ffbcb3e73610fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34765 } }
I20260924 10:13:06.074376 27863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:13:06.074522 27863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:13:06.074693 27863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:06.076830 27863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96381f1a35424b8d8ffbcb3e73610fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34765 } }
I20260924 10:13:06.077306 27863 leader_election.cc:302] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96381f1a35424b8d8ffbcb3e73610fc9; no voters:
I20260924 10:13:06.077967 27863 leader_election.cc:288] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:13:06.078095 27873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:13:06.078907 27873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [term 1 LEADER]: Becoming Leader. State: Replica: 96381f1a35424b8d8ffbcb3e73610fc9, State: Running, Role: LEADER
I20260924 10:13:06.079445 27873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96381f1a35424b8d8ffbcb3e73610fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34765 } }
I20260924 10:13:06.080205 27863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:06.082302 27874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96381f1a35424b8d8ffbcb3e73610fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96381f1a35424b8d8ffbcb3e73610fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34765 } } }
I20260924 10:13:06.082417 27875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96381f1a35424b8d8ffbcb3e73610fc9. Latest consensus state: current_term: 1 leader_uuid: "96381f1a35424b8d8ffbcb3e73610fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96381f1a35424b8d8ffbcb3e73610fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34765 } } }
I20260924 10:13:06.082717 27874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:06.082811 27875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:06.088109 27880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:13:06.093353 27880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:13:06.102255 27880 catalog_manager.cc:1347] Generated new cluster ID: a01434f53a514dd4b74053c0f930ef11
I20260924 10:13:06.102365 27880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:13:06.123023 27880 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:13:06.123504 27880 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:13:06.132030 27880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96381f1a35424b8d8ffbcb3e73610fc9: Generated new TSK 0
I20260924 10:13:06.133042 27880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:13:06.142620 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27796
W20260924 10:13:06.144407 27858 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.211:38611
W20260924 10:13:06.145085 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27796: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:13:06.145238 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 27796
2026-09-24T10:13:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1868 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:13:06 dist-test-slave-s3n3 krb5kdc[27894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:13:06 dist-test-slave-s3n3 krb5kdc[27894](info): set up 2 sockets
Sep 24 10:13:06 dist-test-slave-s3n3 krb5kdc[27894](info): commencing operation
krb5kdc: starting...
W20260924 10:13:07.251606 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:13:07 dist-test-slave-s3n3 krb5kdc[27894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:13:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:13:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:13:07.503433 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:40053
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:13:07.826593 27907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:07.826889 27907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:07.827051 27907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:07.834286 27907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:07.834408 27907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:07.834503 27907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:07.834617 27907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:07.834688 27907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:07.844719 27907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:40053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:07.846107 27907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:07.847453 27907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:07.859525 27912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:07.859565 27915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:07.859503 27913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:07.860404 27907 server_base.cc:1034] running on GCE node
I20260924 10:13:07.861166 27907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:07.862694 27907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:07.863920 27907 hybrid_clock.cc:648] HybridClock initialized: now 1790244787863853 us; error 90 us; skew 500 ppm
I20260924 10:13:07.864271 27907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:07 dist-test-slave-s3n3 krb5kdc[27894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244787, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:07.874704 27907 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:13:07.876410 27907 webserver.cc:460] Webserver started at http://127.6.2.254:45279/ using document root <none> and password file <none>
I20260924 10:13:07.877138 27907 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:07.877249 27907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:07.877528 27907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:07.880903 27907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "37be40a5a0e647bdb2b509c152bccf70"
format_stamp: "Formatted at 2026-09-24 10:13:07 on dist-test-slave-s3n3"
server_key: "c356ba2a5457cffeb8e64b67d39d1263"
server_key_iv: "5f1c5707a3c619e124b9c6336f2a7942"
server_key_version: "encryptionkey@0"
I20260924 10:13:07.881705 27907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "37be40a5a0e647bdb2b509c152bccf70"
format_stamp: "Formatted at 2026-09-24 10:13:07 on dist-test-slave-s3n3"
server_key: "c356ba2a5457cffeb8e64b67d39d1263"
server_key_iv: "5f1c5707a3c619e124b9c6336f2a7942"
server_key_version: "encryptionkey@0"
I20260924 10:13:07.887316 27907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 10:13:07.890923 27922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:07.892396 27907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:13:07.892602 27907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "37be40a5a0e647bdb2b509c152bccf70"
format_stamp: "Formatted at 2026-09-24 10:13:07 on dist-test-slave-s3n3"
server_key: "c356ba2a5457cffeb8e64b67d39d1263"
server_key_iv: "5f1c5707a3c619e124b9c6336f2a7942"
server_key_version: "encryptionkey@0"
I20260924 10:13:07.892803 27907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:07.906836 27907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:07.910950 27907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:07.911269 27907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:07.930583 27907 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:40053
I20260924 10:13:07.930624 27973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:40053 every 8 connection(s)
I20260924 10:13:07.931990 27907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:13:07.935762 27974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:07.938187 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 27907
I20260924 10:13:07.938431 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:13:07.939204 6155 external_mini_cluster.cc:1428] Setting key e97c90007e7de5d492cc614df9b73849
I20260924 10:13:07.948216 27974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70: Bootstrap starting.
I20260924 10:13:07.952011 27974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:07.953436 27974 log.cc:824] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:13:07 dist-test-slave-s3n3 krb5kdc[27894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:13:07.957408 27974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70: No bootstrap required, opened a new log
I20260924 10:13:07.965885 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:07.941128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41619 (local address 127.6.2.254:40053)
0924 10:13:07.941945 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:07.941957 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:07.941998 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:13:07.943934 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:07.943958 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:07.944394 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:07.944739 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:07.946740 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:07.949249 (+ 2509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:07.951204 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:07.951977 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:07.952188 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:07.958156 (+ 5968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:07.958167 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:07.958182 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:07.958225 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:07.961551 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:07.962382 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:07.962388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:07.962395 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:07.962574 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:07.963106 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:07.963112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:07.963120 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:07.963724 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:07.964107 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:07.964579 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 10:13:07.964910 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":396,"threads_started":1}
I20260924 10:13:07.965709 27974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37be40a5a0e647bdb2b509c152bccf70" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40053 } }
I20260924 10:13:07.966130 27974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:13:07.966240 27974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37be40a5a0e647bdb2b509c152bccf70, State: Initialized, Role: FOLLOWER
I20260924 10:13:07.966857 27974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37be40a5a0e647bdb2b509c152bccf70" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40053 } }
I20260924 10:13:07.967134 27974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:13:07.967288 27974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:13:07.967490 27974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:07.970113 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:13:07.970575 27974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37be40a5a0e647bdb2b509c152bccf70" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40053 } }
I20260924 10:13:07.971107 27974 leader_election.cc:302] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37be40a5a0e647bdb2b509c152bccf70; no voters:
I20260924 10:13:07.971776 27974 leader_election.cc:288] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:13:07.971905 27979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:13:07.974319 27974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:07.978859 27979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [term 1 LEADER]: Becoming Leader. State: Replica: 37be40a5a0e647bdb2b509c152bccf70, State: Running, Role: LEADER
I20260924 10:13:07.979662 27979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37be40a5a0e647bdb2b509c152bccf70" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40053 } }
I20260924 10:13:07.983559 27980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37be40a5a0e647bdb2b509c152bccf70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37be40a5a0e647bdb2b509c152bccf70" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40053 } } }
I20260924 10:13:07.984032 27980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:07.984750 27981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37be40a5a0e647bdb2b509c152bccf70. Latest consensus state: current_term: 1 leader_uuid: "37be40a5a0e647bdb2b509c152bccf70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37be40a5a0e647bdb2b509c152bccf70" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40053 } } }
I20260924 10:13:07.985133 27981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70 [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:07.987164 27994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:13:07.993368 27994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:13:07.999670 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:07.980853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39225 (local address 127.6.2.254:40053)
0924 10:13:07.981076 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:07.981085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:07.981876 (+ 791us) server_negotiation.cc:407] Connection header received
0924 10:13:07.982338 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:07.982348 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:07.982437 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:07.982528 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:07.984702 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:07.986304 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:07.988204 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:07.988653 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:07.988808 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:07.992162 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:07.992168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:07.992173 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:07.992212 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:07.996114 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:07.997182 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:07.997187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:07.997192 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:07.997345 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:07.997952 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:07.997958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:07.997961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:07.998380 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:07.998688 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:07.999083 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 10:13:07.999268 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:13:08.003646 27994 catalog_manager.cc:1347] Generated new cluster ID: 697df081ba19411392ea9317989948dc
I20260924 10:13:08.003814 27994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:13:08.015300 27994 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:13:08.016849 27994 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:13:08.027083 27994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37be40a5a0e647bdb2b509c152bccf70: Generated new TSK 0
I20260924 10:13:08.027935 27994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:13:08.040059 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 27907
I20260924 10:13:08.041306 27969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:13:08.132568 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 27907
W20260924 10:13:08.150188 27892 connection.cc:535] client connection to 127.6.2.254:40053 recv error: Network error: failed to read from TLS socket (remote: 127.6.2.254:40053): Connection reset by peer (error 104)
2026-09-24T10:13:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1987 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:13:08 dist-test-slave-s3n3 krb5kdc[28007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:13:08 dist-test-slave-s3n3 krb5kdc[28007](info): set up 2 sockets
Sep 24 10:13:08 dist-test-slave-s3n3 krb5kdc[28007](info): commencing operation
krb5kdc: starting...
W20260924 10:13:09.295640 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:13:09 dist-test-slave-s3n3 krb5kdc[28007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:13:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:13:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:13:09.563048 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:39985
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:39985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:13:09.901330 28020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:09.901639 28020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:09.901732 28020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:09.909509 28020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:09.909633 28020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:09.909682 28020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:09.909740 28020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:09.909785 28020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:09.919909 28020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:39985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:39985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:09.921239 28020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:09.922474 28020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:09.934758 28025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:09.934715 28028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:09.934782 28026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:09.935851 28020 server_base.cc:1034] running on GCE node
I20260924 10:13:09.936678 28020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:09.938138 28020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:09.939366 28020 hybrid_clock.cc:648] HybridClock initialized: now 1790244789939322 us; error 39 us; skew 500 ppm
I20260924 10:13:09.939731 28020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:09 dist-test-slave-s3n3 krb5kdc[28007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244789, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:09.942837 28020 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:13:09.944450 28020 webserver.cc:460] Webserver started at http://127.6.2.254:45133/ using document root <none> and password file <none>
I20260924 10:13:09.945153 28020 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:09.945297 28020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:09.945603 28020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:09.948642 28020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "4766664e2aba44d5addcad0669080e8f"
format_stamp: "Formatted at 2026-09-24 10:13:09 on dist-test-slave-s3n3"
server_key: "7ec16828fb7ebd11d11601b693b3d3a7"
server_key_iv: "a2f42a86d4b223841182bcb2557c85ce"
server_key_version: "encryptionkey@0"
I20260924 10:13:09.949397 28020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "4766664e2aba44d5addcad0669080e8f"
format_stamp: "Formatted at 2026-09-24 10:13:09 on dist-test-slave-s3n3"
server_key: "7ec16828fb7ebd11d11601b693b3d3a7"
server_key_iv: "a2f42a86d4b223841182bcb2557c85ce"
server_key_version: "encryptionkey@0"
I20260924 10:13:09.954947 28020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 10:13:09.958662 28035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:09.960199 28020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:13:09.960402 28020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "4766664e2aba44d5addcad0669080e8f"
format_stamp: "Formatted at 2026-09-24 10:13:09 on dist-test-slave-s3n3"
server_key: "7ec16828fb7ebd11d11601b693b3d3a7"
server_key_iv: "a2f42a86d4b223841182bcb2557c85ce"
server_key_version: "encryptionkey@0"
I20260924 10:13:09.960621 28020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:09.984558 28020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:09.988849 28020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:09.989169 28020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:10.008482 28020 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:39985
I20260924 10:13:10.008507 28086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:39985 every 8 connection(s)
I20260924 10:13:10.010110 28020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:13:10.013903 28087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:10.019418 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 28020
I20260924 10:13:10.019608 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:13:10.020040 6155 external_mini_cluster.cc:1428] Setting key 54eb4202d154973bfb3c2b9cb999f98d
I20260924 10:13:10.025569 28087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f: Bootstrap starting.
I20260924 10:13:10.029708 28087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:10.030772 28087 log.cc:824] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:13:10.034104 28087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f: No bootstrap required, opened a new log
I20260924 10:13:10.036217 28090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:10.022257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58621 (local address 127.6.2.254:39985)
0924 10:13:10.023358 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:10.023374 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:10.023726 (+ 352us) server_negotiation.cc:407] Connection header received
0924 10:13:10.026189 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:10.026217 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:10.026705 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:10.027174 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:10.027726 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:10.029659 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:10.031032 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:10.031927 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:10.032156 (+ 229us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:10.033370 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:10.033381 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:10.033399 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:10.034117 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:10.034215 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:10.034246 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:10.034400 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:10.034759 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 10:13:10.034928 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":494,"threads_started":1}
I20260924 10:13:10.040351 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:13:10.043874 28087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4766664e2aba44d5addcad0669080e8f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39985 } }
I20260924 10:13:10.044515 28087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:13:10.044653 28087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4766664e2aba44d5addcad0669080e8f, State: Initialized, Role: FOLLOWER
I20260924 10:13:10.045356 28087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4766664e2aba44d5addcad0669080e8f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39985 } }
I20260924 10:13:10.045634 28087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:13:10.045820 28087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:13:10.046059 28087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:10.049279 28087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4766664e2aba44d5addcad0669080e8f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39985 } }
I20260924 10:13:10.049858 28087 leader_election.cc:302] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4766664e2aba44d5addcad0669080e8f; no voters:
I20260924 10:13:10.050676 28087 leader_election.cc:288] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:13:10.050781 28097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:13:10.051887 28097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [term 1 LEADER]: Becoming Leader. State: Replica: 4766664e2aba44d5addcad0669080e8f, State: Running, Role: LEADER
I20260924 10:13:10.052484 28097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4766664e2aba44d5addcad0669080e8f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39985 } }
I20260924 10:13:10.053225 28087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:10.053224 28090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:10.045947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48717 (local address 127.6.2.254:39985)
0924 10:13:10.046207 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:10.046213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:10.046573 (+ 360us) server_negotiation.cc:407] Connection header received
0924 10:13:10.046791 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:10.046795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:10.046862 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:10.046945 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:10.047560 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:10.048319 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:10.050160 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:10.050633 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:10.050794 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:10.052256 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:10.052261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:10.052264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:10.052281 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:10.052345 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:10.052361 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:10.052486 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:10.052879 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 10:13:10.052941 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 10:13:10.055423 28098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4766664e2aba44d5addcad0669080e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4766664e2aba44d5addcad0669080e8f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39985 } } }
I20260924 10:13:10.055915 28098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:10.060487 28099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4766664e2aba44d5addcad0669080e8f. Latest consensus state: current_term: 1 leader_uuid: "4766664e2aba44d5addcad0669080e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4766664e2aba44d5addcad0669080e8f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 39985 } } }
I20260924 10:13:10.060933 28099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:10.070981 28102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:13:10.077116 28102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:13:10.085327 28102 catalog_manager.cc:1347] Generated new cluster ID: 61b4f740a8db4b6da7b0522532653637
I20260924 10:13:10.085448 28102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:13:10.109428 28102 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:13:10.111940 28102 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:13:10.124233 28102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4766664e2aba44d5addcad0669080e8f: Generated new TSK 0
I20260924 10:13:10.125176 28102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:13:10.140254 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 28020
W20260924 10:13:10.141420 28082 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58621
W20260924 10:13:10.142171 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 28020: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:13:10.142313 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 28020
W20260924 10:13:10.156845 28005 connection.cc:535] client connection to 127.6.2.254:39985 recv error: Network error: failed to read from TLS socket (remote: 127.6.2.254:39985): Connection reset by peer (error 104)
2026-09-24T10:13:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2008 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:13:10 dist-test-slave-s3n3 krb5kdc[28119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:13:10 dist-test-slave-s3n3 krb5kdc[28119](info): set up 2 sockets
Sep 24 10:13:10 dist-test-slave-s3n3 krb5kdc[28119](info): commencing operation
krb5kdc: starting...
W20260924 10:13:11.251436 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:13:11 dist-test-slave-s3n3 krb5kdc[28119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:13:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:13:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:13:11.510596 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:40913
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:44741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:40913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:13:11.837888 28132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:11.838279 28132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:11.838461 28132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:11.845826 28132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:11.845994 28132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:11.846107 28132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:11.846217 28132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:11.846306 28132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:11.856374 28132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:44741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:40913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:40913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:11.857751 28132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:11.859160 28132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:11.871325 28138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:11.871255 28140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:11.871362 28137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:11.872102 28132 server_base.cc:1034] running on GCE node
I20260924 10:13:11.872877 28132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:11.874306 28132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:11.875537 28132 hybrid_clock.cc:648] HybridClock initialized: now 1790244791875487 us; error 74 us; skew 500 ppm
I20260924 10:13:11.875881 28132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:11 dist-test-slave-s3n3 krb5kdc[28119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244791, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:11.879230 28132 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:13:11.880695 28132 webserver.cc:460] Webserver started at http://127.6.2.254:32995/ using document root <none> and password file <none>
I20260924 10:13:11.881369 28132 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:11.881521 28132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:11.881819 28132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:11.884863 28132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "b886d35919504be7986c6edb410e4947"
format_stamp: "Formatted at 2026-09-24 10:13:11 on dist-test-slave-s3n3"
server_key: "98903b7fdb53e323219c26150ddf65de"
server_key_iv: "809c72ef1bc96dc3f19f4f0a6ef2b11c"
server_key_version: "encryptionkey@0"
I20260924 10:13:11.885632 28132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "b886d35919504be7986c6edb410e4947"
format_stamp: "Formatted at 2026-09-24 10:13:11 on dist-test-slave-s3n3"
server_key: "98903b7fdb53e323219c26150ddf65de"
server_key_iv: "809c72ef1bc96dc3f19f4f0a6ef2b11c"
server_key_version: "encryptionkey@0"
I20260924 10:13:11.890653 28132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:13:11.894177 28147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:11.895558 28132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:13:11.895735 28132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "b886d35919504be7986c6edb410e4947"
format_stamp: "Formatted at 2026-09-24 10:13:11 on dist-test-slave-s3n3"
server_key: "98903b7fdb53e323219c26150ddf65de"
server_key_iv: "809c72ef1bc96dc3f19f4f0a6ef2b11c"
server_key_version: "encryptionkey@0"
I20260924 10:13:11.895931 28132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:11.901616 28132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:11.906025 28132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:11.906327 28132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:11.924971 28132 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:40913
I20260924 10:13:11.925014 28198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:40913 every 8 connection(s)
I20260924 10:13:11.926383 28132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:13:11.930049 28199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:11.934831 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 28132
I20260924 10:13:11.934998 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:13:11.935555 6155 external_mini_cluster.cc:1428] Setting key b2ba1155f179c9090bb60c3f27f54ff4
I20260924 10:13:11.941371 28199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947: Bootstrap starting.
I20260924 10:13:11.943272 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:11.937274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57993 (local address 127.6.2.254:40913)
0924 10:13:11.938087 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:11.938099 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:11.938139 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:13:11.939965 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:11.939987 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:11.940309 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:11.940677 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:11.941710 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:11.941716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:11.941728 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:11.941740 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:11.941790 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:11.941812 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:11.942052 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:11.942185 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 10:13:11.942351 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":344,"threads_started":1}
I20260924 10:13:11.944787 28199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:11.945992 28199 log.cc:824] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947: Log is configured to *not* fsync() on all Append() calls
I20260924 10:13:11.946905 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:13:11.948705 28199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947: No bootstrap required, opened a new log
I20260924 10:13:11.953732 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:11.951372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49205 (local address 127.6.2.254:40913)
0924 10:13:11.951567 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:11.951573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:11.951783 (+ 210us) server_negotiation.cc:407] Connection header received
0924 10:13:11.951976 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:11.951982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:11.952040 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:11.952146 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:11.953086 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:11.953091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:11.953096 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:11.953113 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:11.953163 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:11.953181 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:11.953307 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:11.953389 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 10:13:11.953455 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:13:11.956648 28199 raft_consensus.cc:348] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b886d35919504be7986c6edb410e4947" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40913 } }
I20260924 10:13:11.957053 28199 raft_consensus.cc:374] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:13:11.957166 28199 raft_consensus.cc:729] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b886d35919504be7986c6edb410e4947, State: Initialized, Role: FOLLOWER
I20260924 10:13:11.957705 28199 consensus_queue.cc:260] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b886d35919504be7986c6edb410e4947" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40913 } }
I20260924 10:13:11.957969 28199 raft_consensus.cc:388] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:13:11.958117 28199 raft_consensus.cc:482] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:13:11.958287 28199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:11.960832 28199 raft_consensus.cc:504] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b886d35919504be7986c6edb410e4947" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40913 } }
I20260924 10:13:11.961256 28199 leader_election.cc:302] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b886d35919504be7986c6edb410e4947; no voters:
I20260924 10:13:11.961822 28199 leader_election.cc:288] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:13:11.961928 28209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:13:11.963228 28209 raft_consensus.cc:686] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [term 1 LEADER]: Becoming Leader. State: Replica: b886d35919504be7986c6edb410e4947, State: Running, Role: LEADER
I20260924 10:13:11.963814 28209 consensus_queue.cc:237] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b886d35919504be7986c6edb410e4947" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40913 } }
I20260924 10:13:11.964182 28199 sys_catalog.cc:564] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:11.966606 28211 sys_catalog.cc:455] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b886d35919504be7986c6edb410e4947. Latest consensus state: current_term: 1 leader_uuid: "b886d35919504be7986c6edb410e4947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b886d35919504be7986c6edb410e4947" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40913 } } }
I20260924 10:13:11.966722 28210 sys_catalog.cc:455] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b886d35919504be7986c6edb410e4947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b886d35919504be7986c6edb410e4947" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40913 } } }
I20260924 10:13:11.967020 28211 sys_catalog.cc:458] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:11.967109 28210 sys_catalog.cc:458] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947 [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:11.969691 28216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:13:11.974782 28216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:13:11.983696 28216 catalog_manager.cc:1347] Generated new cluster ID: 903f4ef163d640c78b65e0861de3459b
I20260924 10:13:11.983824 28216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:13:12.002630 28216 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:13:12.003127 28216 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:13:12.014077 28216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b886d35919504be7986c6edb410e4947: Generated new TSK 0
I20260924 10:13:12.015189 28216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:13:12.028882 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 28132
W20260924 10:13:12.029942 28194 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57993
W20260924 10:13:12.030620 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 28132: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:13:12.030764 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 28132
2026-09-24T10:13:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1889 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:13:12 dist-test-slave-s3n3 krb5kdc[28230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:13:12 dist-test-slave-s3n3 krb5kdc[28230](info): set up 2 sockets
Sep 24 10:13:12 dist-test-slave-s3n3 krb5kdc[28230](info): commencing operation
krb5kdc: starting...
W20260924 10:13:13.151597 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:13:13 dist-test-slave-s3n3 krb5kdc[28230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:13:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:13:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:13:13.414902 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:41643
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:41643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:13:13.743779 28243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:13.744128 28243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:13.744302 28243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:13.751588 28243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:13.751744 28243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:13.751840 28243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:13.751955 28243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:13.752058 28243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:13.761981 28243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:41643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:41643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:13.763350 28243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:13.764691 28243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:13.776661 28248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:13.776736 28249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:13.776768 28251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:13.777731 28243 server_base.cc:1034] running on GCE node
I20260924 10:13:13.778509 28243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:13.780005 28243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:13.781217 28243 hybrid_clock.cc:648] HybridClock initialized: now 1790244793781174 us; error 30 us; skew 500 ppm
I20260924 10:13:13.781577 28243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:13 dist-test-slave-s3n3 krb5kdc[28230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244793, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:13.790107 28243 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:13:13.791826 28243 webserver.cc:460] Webserver started at http://127.6.2.254:41237/ using document root <none> and password file <none>
I20260924 10:13:13.792549 28243 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:13.792701 28243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:13.793006 28243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:13.796094 28243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "85d48cf2b3ab4312bbaecafa39898fca"
format_stamp: "Formatted at 2026-09-24 10:13:13 on dist-test-slave-s3n3"
server_key: "16c4980027a8fbbeab3c559ffe13deec"
server_key_iv: "9030335ff6c28e25aeddb1c533e15eef"
server_key_version: "encryptionkey@0"
I20260924 10:13:13.796868 28243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "85d48cf2b3ab4312bbaecafa39898fca"
format_stamp: "Formatted at 2026-09-24 10:13:13 on dist-test-slave-s3n3"
server_key: "16c4980027a8fbbeab3c559ffe13deec"
server_key_iv: "9030335ff6c28e25aeddb1c533e15eef"
server_key_version: "encryptionkey@0"
I20260924 10:13:13.802130 28243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:13:13.805524 28258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:13.806797 28243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:13:13.806979 28243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "85d48cf2b3ab4312bbaecafa39898fca"
format_stamp: "Formatted at 2026-09-24 10:13:13 on dist-test-slave-s3n3"
server_key: "16c4980027a8fbbeab3c559ffe13deec"
server_key_iv: "9030335ff6c28e25aeddb1c533e15eef"
server_key_version: "encryptionkey@0"
I20260924 10:13:13.807242 28243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:13.825655 28243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:13.830408 28243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:13.830760 28243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:13.850669 28243 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:41643
I20260924 10:13:13.850692 28309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:41643 every 8 connection(s)
I20260924 10:13:13.852231 28243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:13:13.855988 28310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:13.859613 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 28243
I20260924 10:13:13.859819 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:13:13.860266 6155 external_mini_cluster.cc:1428] Setting key 3ceeb22a0d82d19481167fb5d439f4c6
I20260924 10:13:13.869194 28310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca: Bootstrap starting.
I20260924 10:13:13.873163 28310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:13.874478 28310 log.cc:824] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:13:13 dist-test-slave-s3n3 krb5kdc[28230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:13:13.877779 28310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca: No bootstrap required, opened a new log
I20260924 10:13:13.884394 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:13.862182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37117 (local address 127.6.2.254:41643)
0924 10:13:13.863011 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:13.863022 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:13.863119 (+ 97us) server_negotiation.cc:407] Connection header received
0924 10:13:13.864896 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:13.864920 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:13.865372 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:13.865724 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:13.867581 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:13.869111 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:13.870966 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:13.871744 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:13.871883 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:13.876848 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:13.876856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:13.876870 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:13.876915 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:13.880137 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:13.880824 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:13.880831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:13.880840 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:13.881008 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:13.881389 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:13.881394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:13.881397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:13.882043 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:13.882357 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:13.882826 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 10:13:13.883287 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":376,"threads_started":1}
I20260924 10:13:13.887075 28310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d48cf2b3ab4312bbaecafa39898fca" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41643 } }
I20260924 10:13:13.887518 28310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:13:13.887625 28310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d48cf2b3ab4312bbaecafa39898fca, State: Initialized, Role: FOLLOWER
I20260924 10:13:13.888242 28310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d48cf2b3ab4312bbaecafa39898fca" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41643 } }
I20260924 10:13:13.888563 28310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:13:13.888741 28310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:13:13.888942 28310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:13.889806 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:13:13.891417 28310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d48cf2b3ab4312bbaecafa39898fca" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41643 } }
I20260924 10:13:13.891913 28310 leader_election.cc:302] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85d48cf2b3ab4312bbaecafa39898fca; no voters:
I20260924 10:13:13.892535 28310 leader_election.cc:288] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:13:13.892761 28315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:13:13.894611 28310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:13.898566 28315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [term 1 LEADER]: Becoming Leader. State: Replica: 85d48cf2b3ab4312bbaecafa39898fca, State: Running, Role: LEADER
I20260924 10:13:13.899549 28315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d48cf2b3ab4312bbaecafa39898fca" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41643 } }
I20260924 10:13:13.902884 28317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85d48cf2b3ab4312bbaecafa39898fca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d48cf2b3ab4312bbaecafa39898fca" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41643 } } }
I20260924 10:13:13.903296 28317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:13.903906 28318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85d48cf2b3ab4312bbaecafa39898fca. Latest consensus state: current_term: 1 leader_uuid: "85d48cf2b3ab4312bbaecafa39898fca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d48cf2b3ab4312bbaecafa39898fca" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 41643 } } }
I20260924 10:13:13.904258 28318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:13.913857 28330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:13:13.919513 28330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:13:13.925678 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:13.902609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52413 (local address 127.6.2.254:41643)
0924 10:13:13.902819 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:13.902826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:13.905964 (+ 3138us) server_negotiation.cc:407] Connection header received
0924 10:13:13.906890 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:13.906900 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:13.907024 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:13.907183 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:13.909833 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:13.911357 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:13.913846 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:13.914863 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:13.914998 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:13.918244 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:13.918253 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:13.918260 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:13.918307 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:13.922279 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:13.923172 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:13.923178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:13.923181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:13.923330 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:13.923803 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:13.923808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:13.923811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:13.924258 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:13.924497 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:13.925071 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 10:13:13.925310 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:13:13.931421 28330 catalog_manager.cc:1347] Generated new cluster ID: 69f3cfbba0634427a73cd0c2d517b31c
I20260924 10:13:13.931593 28330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:13:13.946043 28330 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:13:13.947438 28330 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:13:13.954250 28330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85d48cf2b3ab4312bbaecafa39898fca: Generated new TSK 0
I20260924 10:13:13.955204 28330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:13:13.962908 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 28243
I20260924 10:13:13.963984 28305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:13:14.054550 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 28243
2026-09-24T10:13:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2023 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:13:14 dist-test-slave-s3n3 krb5kdc[28343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:13:14 dist-test-slave-s3n3 krb5kdc[28343](info): set up 2 sockets
Sep 24 10:13:14 dist-test-slave-s3n3 krb5kdc[28343](info): commencing operation
krb5kdc: starting...
W20260924 10:13:15.163552 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:13:15 dist-test-slave-s3n3 krb5kdc[28343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:13:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:13:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:13:15.418056 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36585
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:36585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:13:15.744827 28356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:15.745173 28356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:15.745348 28356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:15.752719 28356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:15.752888 28356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:15.752995 28356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:15.753110 28356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:15.753196 28356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:15.763438 28356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:36585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:36585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:15.764798 28356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:15.766103 28356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:15.777904 28361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:15.777866 28364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:15.778203 28362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:15.779014 28356 server_base.cc:1034] running on GCE node
I20260924 10:13:15.779804 28356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:15.781154 28356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:15.782397 28356 hybrid_clock.cc:648] HybridClock initialized: now 1790244795782354 us; error 55 us; skew 500 ppm
I20260924 10:13:15.782753 28356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:15 dist-test-slave-s3n3 krb5kdc[28343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244795, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:15.786029 28356 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:13:15.787552 28356 webserver.cc:460] Webserver started at http://127.6.2.254:45415/ using document root <none> and password file <none>
I20260924 10:13:15.788241 28356 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:15.788385 28356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:15.788687 28356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:15.791759 28356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "5065877f5d214336a2be0a921ce1e5c4"
format_stamp: "Formatted at 2026-09-24 10:13:15 on dist-test-slave-s3n3"
server_key: "98aa55703b711df0489fa456137c7167"
server_key_iv: "b653e938e0bbc0d3091d1d586e145270"
server_key_version: "encryptionkey@0"
I20260924 10:13:15.792512 28356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "5065877f5d214336a2be0a921ce1e5c4"
format_stamp: "Formatted at 2026-09-24 10:13:15 on dist-test-slave-s3n3"
server_key: "98aa55703b711df0489fa456137c7167"
server_key_iv: "b653e938e0bbc0d3091d1d586e145270"
server_key_version: "encryptionkey@0"
I20260924 10:13:15.797748 28356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:13:15.801766 28371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:15.803025 28356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:13:15.803246 28356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "5065877f5d214336a2be0a921ce1e5c4"
format_stamp: "Formatted at 2026-09-24 10:13:15 on dist-test-slave-s3n3"
server_key: "98aa55703b711df0489fa456137c7167"
server_key_iv: "b653e938e0bbc0d3091d1d586e145270"
server_key_version: "encryptionkey@0"
I20260924 10:13:15.803457 28356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:15.818988 28356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:15.823489 28356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:15.823767 28356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:15.842617 28356 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:36585
I20260924 10:13:15.842646 28422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:36585 every 8 connection(s)
I20260924 10:13:15.844031 28356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:13:15.847750 28423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:15.851883 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 28356
I20260924 10:13:15.852064 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:13:15.852643 6155 external_mini_cluster.cc:1428] Setting key b2807f5a115b37da62b58e7c39565b4d
I20260924 10:13:15.861275 28423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4: Bootstrap starting.
I20260924 10:13:15.865057 28423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:15.866129 28423 log.cc:824] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4: Log is configured to *not* fsync() on all Append() calls
I20260924 10:13:15.866310 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:15.854394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59495 (local address 127.6.2.254:36585)
0924 10:13:15.855207 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:15.855218 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:15.855256 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:13:15.856844 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:15.856864 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:15.857266 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:15.857585 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:15.858169 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:15.860067 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:15.861825 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:15.862447 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:15.862598 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:15.863912 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:15.863920 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:15.863932 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:15.864426 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:15.864499 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:15.864531 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:15.864613 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:15.864952 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 10:13:15.865266 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":391,"threads_started":1}
I20260924 10:13:15.869082 28423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4: No bootstrap required, opened a new log
I20260924 10:13:15.870095 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:13:15.877163 28423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5065877f5d214336a2be0a921ce1e5c4" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36585 } }
I20260924 10:13:15.877554 28423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:13:15.877676 28423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5065877f5d214336a2be0a921ce1e5c4, State: Initialized, Role: FOLLOWER
I20260924 10:13:15.878306 28423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5065877f5d214336a2be0a921ce1e5c4" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36585 } }
I20260924 10:13:15.878568 28423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:13:15.878744 28423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:13:15.878916 28423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:15.880506 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:15.874377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58081 (local address 127.6.2.254:36585)
0924 10:13:15.874556 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:15.874565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:15.875018 (+ 453us) server_negotiation.cc:407] Connection header received
0924 10:13:15.875212 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:15.875217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:15.875284 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:15.875398 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:15.875898 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:15.876637 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:15.878028 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:15.878461 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:15.878592 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:15.879805 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:15.879810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:15.879814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:15.879831 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:15.879870 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:15.879884 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:15.879996 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:15.880122 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 10:13:15.880213 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:13:15.881454 28423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5065877f5d214336a2be0a921ce1e5c4" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36585 } }
I20260924 10:13:15.882001 28423 leader_election.cc:302] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5065877f5d214336a2be0a921ce1e5c4; no voters:
I20260924 10:13:15.882534 28423 leader_election.cc:288] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:13:15.882673 28433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:13:15.883598 28433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5065877f5d214336a2be0a921ce1e5c4, State: Running, Role: LEADER
I20260924 10:13:15.884119 28433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5065877f5d214336a2be0a921ce1e5c4" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36585 } }
I20260924 10:13:15.884961 28423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:15.887230 28434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5065877f5d214336a2be0a921ce1e5c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5065877f5d214336a2be0a921ce1e5c4" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36585 } } }
I20260924 10:13:15.887275 28435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5065877f5d214336a2be0a921ce1e5c4. Latest consensus state: current_term: 1 leader_uuid: "5065877f5d214336a2be0a921ce1e5c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5065877f5d214336a2be0a921ce1e5c4" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 36585 } } }
I20260924 10:13:15.887588 28434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:15.887645 28435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4 [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:15.888980 28438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:13:15.893730 28438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:13:15.903790 28438 catalog_manager.cc:1347] Generated new cluster ID: c4ce2674bf0c4d798e58a4895ea704c3
I20260924 10:13:15.903915 28438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:13:15.927805 28438 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:13:15.929618 28438 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:13:15.936964 28438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5065877f5d214336a2be0a921ce1e5c4: Generated new TSK 0
I20260924 10:13:15.937759 28438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:13:15.951159 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 28356
W20260924 10:13:15.952453 28418 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59495
W20260924 10:13:15.953154 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 28356: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:13:15.953307 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 28356
2026-09-24T10:13:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1898 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:13:16 dist-test-slave-s3n3 krb5kdc[28454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:13:16 dist-test-slave-s3n3 krb5kdc[28454](info): set up 2 sockets
Sep 24 10:13:16 dist-test-slave-s3n3 krb5kdc[28454](info): commencing operation
krb5kdc: starting...
W20260924 10:13:17.067605 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:13:17 dist-test-slave-s3n3 krb5kdc[28454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:13:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:13:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:13:17.330931 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:43711
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:43711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:13:17.666329 28467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:17.666641 28467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:17.666798 28467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:17.674404 28467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:17.674535 28467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:17.674621 28467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:17.674796 28467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:17.674854 28467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:17.685278 28467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:43711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:43711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:17.686632 28467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:17.687985 28467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:17.699853 28475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:17.699888 28473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:17.699918 28472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:17.700687 28467 server_base.cc:1034] running on GCE node
I20260924 10:13:17.701488 28467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:17.702931 28467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:17.704136 28467 hybrid_clock.cc:648] HybridClock initialized: now 1790244797704090 us; error 37 us; skew 500 ppm
I20260924 10:13:17.704502 28467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:17 dist-test-slave-s3n3 krb5kdc[28454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:17.707620 28467 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:13:17.709185 28467 webserver.cc:460] Webserver started at http://127.6.2.254:43317/ using document root <none> and password file <none>
I20260924 10:13:17.709877 28467 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:17.710031 28467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:17.710340 28467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:17.713387 28467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "52a523d92d3945e5bd8140ce556e471a"
format_stamp: "Formatted at 2026-09-24 10:13:17 on dist-test-slave-s3n3"
server_key: "fd1f229bde3dc0c0db467314ffbc06aa"
server_key_iv: "4fbf865eaaf58468b70636497bee0ea7"
server_key_version: "encryptionkey@0"
I20260924 10:13:17.714169 28467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "52a523d92d3945e5bd8140ce556e471a"
format_stamp: "Formatted at 2026-09-24 10:13:17 on dist-test-slave-s3n3"
server_key: "fd1f229bde3dc0c0db467314ffbc06aa"
server_key_iv: "4fbf865eaaf58468b70636497bee0ea7"
server_key_version: "encryptionkey@0"
I20260924 10:13:17.719511 28467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 10:13:17.723150 28482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:17.724545 28467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:13:17.724748 28467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "52a523d92d3945e5bd8140ce556e471a"
format_stamp: "Formatted at 2026-09-24 10:13:17 on dist-test-slave-s3n3"
server_key: "fd1f229bde3dc0c0db467314ffbc06aa"
server_key_iv: "4fbf865eaaf58468b70636497bee0ea7"
server_key_version: "encryptionkey@0"
I20260924 10:13:17.724963 28467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:17.730907 28467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:17.735430 28467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:17.735764 28467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:17.754755 28467 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:43711
I20260924 10:13:17.754839 28533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:43711 every 8 connection(s)
I20260924 10:13:17.756224 28467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:13:17.756510 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 28467
I20260924 10:13:17.756698 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:13:17.757143 6155 external_mini_cluster.cc:1428] Setting key d73508b1f417eaeaf16c593ed5962c80
I20260924 10:13:17.761160 28534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:17.766371 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:17.759182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33065 (local address 127.6.2.254:43711)
0924 10:13:17.759970 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:17.759986 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:17.760041 (+ 55us) server_negotiation.cc:407] Connection header received
0924 10:13:17.762447 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:17.762505 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:17.762933 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:17.763379 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:17.764399 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:17.764408 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:17.764436 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:17.764460 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:17.764552 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:17.764578 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:17.764796 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:17.764962 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 10:13:17.765147 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":326,"threads_started":1}
I20260924 10:13:17.770548 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:13:17.775614 28534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a: Bootstrap starting.
I20260924 10:13:17.778566 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:17.775883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44573 (local address 127.6.2.254:43711)
0924 10:13:17.776116 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:17.776125 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:17.776554 (+ 429us) server_negotiation.cc:407] Connection header received
0924 10:13:17.776736 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:17.776741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:17.776790 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:17.776887 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:17.777855 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:17.777859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:17.777862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:13:17.777905 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:17.777941 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:13:17.777957 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:17.778067 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:17.778198 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 10:13:17.778244 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:13:17.779798 28534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:17.780814 28534 log.cc:824] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:13:17.783767 28534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a: No bootstrap required, opened a new log
I20260924 10:13:17.791177 28534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a523d92d3945e5bd8140ce556e471a" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 43711 } }
I20260924 10:13:17.791653 28534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:13:17.791785 28534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a523d92d3945e5bd8140ce556e471a, State: Initialized, Role: FOLLOWER
I20260924 10:13:17.792356 28534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a523d92d3945e5bd8140ce556e471a" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 43711 } }
I20260924 10:13:17.792590 28534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:13:17.792742 28534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:13:17.792932 28534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:17.795114 28534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a523d92d3945e5bd8140ce556e471a" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 43711 } }
I20260924 10:13:17.795625 28534 leader_election.cc:302] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52a523d92d3945e5bd8140ce556e471a; no voters:
I20260924 10:13:17.796252 28534 leader_election.cc:288] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:13:17.796406 28544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:13:17.797587 28544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [term 1 LEADER]: Becoming Leader. State: Replica: 52a523d92d3945e5bd8140ce556e471a, State: Running, Role: LEADER
I20260924 10:13:17.798198 28544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a523d92d3945e5bd8140ce556e471a" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 43711 } }
I20260924 10:13:17.798555 28534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:17.800860 28545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52a523d92d3945e5bd8140ce556e471a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a523d92d3945e5bd8140ce556e471a" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 43711 } } }
I20260924 10:13:17.801257 28545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:17.801770 28546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52a523d92d3945e5bd8140ce556e471a. Latest consensus state: current_term: 1 leader_uuid: "52a523d92d3945e5bd8140ce556e471a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a523d92d3945e5bd8140ce556e471a" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 43711 } } }
I20260924 10:13:17.802100 28546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:17.802232 28551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:13:17.824038 28551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:13:17.833446 28551 catalog_manager.cc:1347] Generated new cluster ID: a2874a00b716438eabac019800eab765
I20260924 10:13:17.833585 28551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:13:17.852734 28551 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:13:17.853334 28551 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:13:17.870059 28551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52a523d92d3945e5bd8140ce556e471a: Generated new TSK 0
I20260924 10:13:17.871155 28551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:13:17.884038 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 28467
W20260924 10:13:17.885170 28529 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33065
W20260924 10:13:17.885912 6155 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 28467: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 10:13:17.886070 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 28467
2026-09-24T10:13:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1932 ms)
[----------] 18 tests from AuthTokenIssuingTest (35082 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:13:17 dist-test-slave-s3n3 krb5kdc[28566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:13:17 dist-test-slave-s3n3 krb5kdc[28566](info): set up 2 sockets
Sep 24 10:13:17 dist-test-slave-s3n3 krb5kdc[28566](info): commencing operation
krb5kdc: starting...
W20260924 10:13:19.005318 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:13:19 dist-test-slave-s3n3 krb5kdc[28566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:13:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:13:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:13:19.275719 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:42313
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:42313,127.6.2.253:41159,127.6.2.252:35299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:13:19.601504 28579 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:13:19.601855 28579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:19.601958 28579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:19.602092 28579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:19.609688 28579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:19.609798 28579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:19.609894 28579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:19.609974 28579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:19.610087 28579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:19.620317 28579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:42313,127.6.2.253:41159,127.6.2.252:35299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:42313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:19.621737 28579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:19.623190 28579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:19.634832 28587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:19.634876 28584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:19.634848 28585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:19.636162 28579 server_base.cc:1034] running on GCE node
I20260924 10:13:19.636919 28579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:19.638350 28579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:19.639595 28579 hybrid_clock.cc:648] HybridClock initialized: now 1790244799639543 us; error 56 us; skew 500 ppm
I20260924 10:13:19.639919 28579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:19 dist-test-slave-s3n3 krb5kdc[28566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:19.642933 28579 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:13:19.644465 28579 webserver.cc:460] Webserver started at http://127.6.2.254:35465/ using document root <none> and password file <none>
I20260924 10:13:19.645146 28579 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:19.645277 28579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:19.645599 28579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:19.648617 28579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "ebc117e058604c1f8dfa46743680ef72"
format_stamp: "Formatted at 2026-09-24 10:13:19 on dist-test-slave-s3n3"
server_key: "1e7fd5b56c1eb61d73f6dba59e417ee8"
server_key_iv: "895b5434386aa6c7ac7bea81b8b91e48"
server_key_version: "encryptionkey@0"
I20260924 10:13:19.649420 28579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "ebc117e058604c1f8dfa46743680ef72"
format_stamp: "Formatted at 2026-09-24 10:13:19 on dist-test-slave-s3n3"
server_key: "1e7fd5b56c1eb61d73f6dba59e417ee8"
server_key_iv: "895b5434386aa6c7ac7bea81b8b91e48"
server_key_version: "encryptionkey@0"
I20260924 10:13:19.654521 28579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 10:13:19.657955 28594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:19.659267 28579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:13:19.659447 28579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "ebc117e058604c1f8dfa46743680ef72"
format_stamp: "Formatted at 2026-09-24 10:13:19 on dist-test-slave-s3n3"
server_key: "1e7fd5b56c1eb61d73f6dba59e417ee8"
server_key_iv: "895b5434386aa6c7ac7bea81b8b91e48"
server_key_version: "encryptionkey@0"
I20260924 10:13:19.659675 28579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:19.678952 28579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:19.683122 28579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:19.683436 28579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:19.702073 28579 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:42313
I20260924 10:13:19.702092 28645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:42313 every 8 connection(s)
I20260924 10:13:19.703462 28579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:13:19.707630 28646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:19.711175 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 28579
I20260924 10:13:19.711407 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:13:19.712236 6155 external_mini_cluster.cc:1428] Setting key 3455ff9f46349c3759dcf18fb46b54c2
I20260924 10:13:19.716390 28646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:13:19 dist-test-slave-s3n3 krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.2.253@KRBTEST.COM; defaulting to no policy
I20260924 10:13:19.745433 28649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:19.721808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58110 (local address 127.6.2.254:42313)
0924 10:13:19.723254 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:19.723265 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:19.723322 (+ 57us) server_negotiation.cc:407] Connection header received
0924 10:13:19.724636 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:19.724645 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:19.725035 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:19.725285 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:19.727735 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:19.729005 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:19.731378 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:19.732139 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:19.732284 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:19.736830 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:19.736839 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:19.736852 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:19.736882 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:19.740117 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:19.741152 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:19.741159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:19.741162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:19.741332 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:19.741999 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:19.742006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:19.742008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:19.742726 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:19.742905 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:19.743746 (+ 841us) server_negotiation.cc:299] Negotiation successful
0924 10:13:19.744454 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":272,"server-negotiator.queue_time_us":1200,"thread_start_us":1007,"threads_started":1}
I20260924 10:13:19.745502 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:19.720423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:42313 (local address 127.0.0.1:58110)
0924 10:13:19.721333 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:19.721387 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:19.724059 (+ 2672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:19.725401 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:19.725411 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:19.726470 (+ 1059us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:19.727419 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:19.727459 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:19.729235 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:19.729243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:19.731209 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:19.731228 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:19.732382 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:19.732391 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:19.732633 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:19.733362 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:19.733397 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:19.736534 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:19.740280 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:19.740297 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:19.740314 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:19.740973 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:19.741465 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:19.741472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:19.741476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:19.741712 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:19.742941 (+ 1229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:19.742953 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:19.743314 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 10:13:19.743573 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 10:13:19.744079 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":393,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.253@KRBTEST.COM" created.
I20260924 10:13:19.748777 28646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:13:19.750269 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:19.749733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:41159 (local address 127.0.0.1:36150)
0924 10:13:19.749967 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:19.750127 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260924 10:13:19.750643 28595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
I20260924 10:13:19.752813 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:19.752362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:41159 (local address 127.0.0.1:36156)
0924 10:13:19.752559 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:19.752654 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 10:13:19.753041 28646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:41159: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.253.keytab.
Entry for principal kudu/127.6.2.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.253.keytab.
I20260924 10:13:19.780741 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.253:41159
--webserver_interface=127.6.2.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.253.keytab
--principal=kudu/127.6.2.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:42313,127.6.2.253:41159,127.6.2.252:35299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:13:19.802400 28646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } attempt: 1
I20260924 10:13:19.803858 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:19.803281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:41159 (local address 127.0.0.1:36170)
0924 10:13:19.803519 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:19.803653 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 10:13:19.805059 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:19.804618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:41159 (local address 127.0.0.1:36174)
0924 10:13:19.804787 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:19.804887 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 10:13:19.805308 28646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:41159: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
I20260924 10:13:19.873608 28646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } attempt: 2
I20260924 10:13:19.874936 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:19.874371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:41159 (local address 127.0.0.1:36190)
0924 10:13:19.874650 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:19.874739 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 10:13:19.876099 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:19.875691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:41159 (local address 127.0.0.1:36196)
0924 10:13:19.875901 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:19.875983 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 10:13:19.876381 28646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:41159: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
I20260924 10:13:19.967701 28646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } attempt: 3
I20260924 10:13:19.968957 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:19.968471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:41159 (local address 127.0.0.1:36202)
0924 10:13:19.968690 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:19.968789 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 10:13:19.970194 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:19.969687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:41159 (local address 127.0.0.1:36218)
0924 10:13:19.969850 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:19.969920 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 10:13:19.970453 28646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:41159: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
I20260924 10:13:20.113768 28646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } attempt: 4
I20260924 10:13:20.115196 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.114577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:41159 (local address 127.0.0.1:36226)
0924 10:13:20.114880 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.114966 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260924 10:13:20.116331 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.115971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:41159 (local address 127.0.0.1:36232)
0924 10:13:20.116144 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.116221 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260924 10:13:20.116568 28646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:41159: Network error: Client connection negotiation failed: client connection to 127.6.2.253:41159: connect: Connection refused (error 111)
W20260924 10:13:20.128008 28653 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:13:20.128381 28653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:20.128510 28653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:20.128643 28653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:20.136379 28653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:20.136476 28653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:20.136574 28653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:20.136646 28653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:20.136731 28653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:20.146780 28653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:42313,127.6.2.253:41159,127.6.2.252:35299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.253.keytab
--principal=kudu/127.6.2.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.253:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.2.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:20.148182 28653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:20.149503 28653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:20.161180 28658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:20.161217 28659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:20.161343 28661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:20.162101 28653 server_base.cc:1034] running on GCE node
I20260924 10:13:20.162909 28653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:20.164418 28653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:20.165654 28653 hybrid_clock.cc:648] HybridClock initialized: now 1790244800165593 us; error 31 us; skew 500 ppm
I20260924 10:13:20.166013 28653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:20 dist-test-slave-s3n3 krb5kdc[28566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244800, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:20.169651 28653 init.cc:377] Logged in from keytab as kudu/127.6.2.253@KRBTEST.COM (short username kudu)
I20260924 10:13:20.172096 28653 webserver.cc:460] Webserver started at http://127.6.2.253:41773/ using document root <none> and password file <none>
I20260924 10:13:20.173157 28653 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:20.173290 28653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:20.173710 28653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:20.177956 28653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/data/instance:
uuid: "f9e04f65cfe2425b8cef3b7293d5cb96"
format_stamp: "Formatted at 2026-09-24 10:13:20 on dist-test-slave-s3n3"
server_key: "688f3b65dd5508b04e9168be0bf5200c"
server_key_iv: "d5740e8f475e1076317cddc239b5cd88"
server_key_version: "encryptionkey@0"
I20260924 10:13:20.178807 28653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/wal/instance:
uuid: "f9e04f65cfe2425b8cef3b7293d5cb96"
format_stamp: "Formatted at 2026-09-24 10:13:20 on dist-test-slave-s3n3"
server_key: "688f3b65dd5508b04e9168be0bf5200c"
server_key_iv: "d5740e8f475e1076317cddc239b5cd88"
server_key_version: "encryptionkey@0"
I20260924 10:13:20.184224 28653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:13:20.187847 28668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:20.189262 28653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:13:20.189450 28653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/wal
uuid: "f9e04f65cfe2425b8cef3b7293d5cb96"
format_stamp: "Formatted at 2026-09-24 10:13:20 on dist-test-slave-s3n3"
server_key: "688f3b65dd5508b04e9168be0bf5200c"
server_key_iv: "d5740e8f475e1076317cddc239b5cd88"
server_key_version: "encryptionkey@0"
I20260924 10:13:20.189678 28653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:20.216079 28653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:20.220706 28653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:20.221011 28653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:20.240438 28653 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.253:41159
I20260924 10:13:20.240446 28719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.253:41159 every 8 connection(s)
I20260924 10:13:20.241864 28653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/data/info.pb
I20260924 10:13:20.245682 28720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:20.249408 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 28653
I20260924 10:13:20.249894 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-1/wal/instance
I20260924 10:13:20.250682 6155 external_mini_cluster.cc:1428] Setting key 42a5114ff77f229a64bb429421df0a26
I20260924 10:13:20.252794 28720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:13:20 dist-test-slave-s3n3 krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244800, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.253@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:13:20.277225 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.255896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.6.2.254:42313)
0924 10:13:20.256420 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.256425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.257540 (+ 1115us) server_negotiation.cc:407] Connection header received
0924 10:13:20.260387 (+ 2847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.260393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.260520 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.260666 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.262284 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.263136 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.264779 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.265211 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.265353 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.269465 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:20.269474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:20.269482 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:20.269544 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:20.273551 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.274411 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.274419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.274425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.274590 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.275085 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.275093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.275098 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.275522 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:20.275702 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:20.276514 (+ 812us) server_negotiation.cc:299] Negotiation successful
0924 10:13:20.276762 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":310,"threads_started":1}
WARNING: no policy specified for kudu/127.6.2.252@KRBTEST.COM; defaulting to no policy
I20260924 10:13:20.278013 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.256529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:42313 (local address 127.0.0.1:58114)
0924 10:13:20.257377 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.257419 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:20.260111 (+ 2692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:20.260955 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:20.260970 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:20.261493 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:20.262074 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.262092 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.263289 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.263295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.264633 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.264645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.265446 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.265453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.265697 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.266452 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:20.266476 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:20.269186 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:20.273742 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.273750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.273765 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.274258 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.274747 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.274751 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.274754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.274937 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.275687 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:20.275694 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:20.276119 (+ 425us) client_negotiation.cc:769] Sending connection context
0924 10:13:20.276371 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 10:13:20.276781 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":425,"threads_started":1}
I20260924 10:13:20.280087 28720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.252@KRBTEST.COM" created.
Sep 24 10:13:20 dist-test-slave-s3n3 krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244800, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.253@KRBTEST.COM for kudu/127.6.2.253@KRBTEST.COM
I20260924 10:13:20.294955 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.280932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:41159 (local address 127.0.0.1:36238)
0924 10:13:20.281114 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.281130 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:20.281217 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:20.282322 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:20.282327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:20.282768 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:20.282899 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.282907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.284297 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.284303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.285142 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.285153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.285986 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.285991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.286210 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.286929 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:20.286948 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:20.289068 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:20.292376 (+ 3308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.292384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.292389 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.292753 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.293126 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.293132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.293135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.293281 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.293895 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:20.293900 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:20.294067 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 10:13:20.294221 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 10:13:20.294504 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":4}
I20260924 10:13:20.295037 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.281217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.6.2.253:41159)
0924 10:13:20.281880 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.281886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.281912 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:13:20.281956 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.281962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.282158 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.282339 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.283100 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.284138 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.285293 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.285835 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.285973 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.289281 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:20.289293 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:20.289306 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:20.289333 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:20.292217 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.292889 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.292896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.292902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.293016 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.293383 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.293397 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.293400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.293762 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:20.294018 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:20.294360 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 10:13:20.294620 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":522,"thread_start_us":397,"threads_started":1}
I20260924 10:13:20.297676 28720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:13:20.299042 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.298346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:47920)
0924 10:13:20.298644 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.298815 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260924 10:13:20.299350 28669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
I20260924 10:13:20.301012 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.300666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:47934)
0924 10:13:20.300816 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.300902 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 10:13:20.301285 28720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:35299: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.252.keytab.
Entry for principal kudu/127.6.2.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.252.keytab.
I20260924 10:13:20.315752 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.252:35299
--webserver_interface=127.6.2.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.252.keytab
--principal=kudu/127.6.2.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:42313,127.6.2.253:41159,127.6.2.252:35299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:13:20.350618 28720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } attempt: 1
I20260924 10:13:20.351869 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.351405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:47940)
0924 10:13:20.351605 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.351700 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 10:13:20.352864 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.352516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:47942)
0924 10:13:20.352688 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.352758 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:13:20.353091 28720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:35299: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
I20260924 10:13:20.415848 28646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } attempt: 5
I20260924 10:13:20.421314 28720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } attempt: 2
I20260924 10:13:20.422401 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.421900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:47950)
0924 10:13:20.422114 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.422218 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 10:13:20.423524 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.423166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:47966)
0924 10:13:20.423359 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.423417 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260924 10:13:20.423707 28720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:35299: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Sep 24 10:13:20 dist-test-slave-s3n3 krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for kudu/127.6.2.253@KRBTEST.COM
I20260924 10:13:20.430358 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.416907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.6.2.253:41159)
0924 10:13:20.417154 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.417160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.417187 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:13:20.417258 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.417261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.417323 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.417434 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.418237 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.418998 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.420320 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.420770 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.420938 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.424835 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:20.424840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:20.424843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:20.424874 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:20.427568 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.428465 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.428469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.428472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.428589 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.429050 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.429053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.429055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.429322 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:20.429431 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:20.429954 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 10:13:20.430151 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 10:13:20.430577 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.416662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:41159 (local address 127.0.0.1:36252)
0924 10:13:20.416861 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.416888 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:20.417023 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:20.417494 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:20.417500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:20.417905 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:20.418081 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.418089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.419269 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.419275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.420146 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.420157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.420957 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.420963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.421164 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.421857 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:20.421875 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:20.424650 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:20.427708 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.427714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.427718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.428322 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.428711 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.428718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.428723 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.428922 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.429457 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:20.429466 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:20.429667 (+ 201us) client_negotiation.cc:769] Sending connection context
0924 10:13:20.429871 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 10:13:20.430229 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 10:13:20.431723 28646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:13:20.432708 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.432282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:47980)
0924 10:13:20.432462 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.432554 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 10:13:20.433663 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.433289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:47986)
0924 10:13:20.433433 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.433551 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 10:13:20.433910 28646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:35299: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
I20260924 10:13:20.485203 28646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } attempt: 1
I20260924 10:13:20.486644 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.486039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:47998)
0924 10:13:20.486283 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.486437 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 10:13:20.487879 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.487475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:48000)
0924 10:13:20.487647 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.487740 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 10:13:20.488143 28646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:35299: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
I20260924 10:13:20.514973 28720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } attempt: 3
I20260924 10:13:20.516194 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.515719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:48006)
0924 10:13:20.515915 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.516020 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 10:13:20.517390 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.516978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:48008)
0924 10:13:20.517208 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.517271 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260924 10:13:20.517596 28720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:35299: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
I20260924 10:13:20.556396 28646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } attempt: 2
I20260924 10:13:20.557792 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.557186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:48024)
0924 10:13:20.557453 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.557578 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 10:13:20.559000 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.558574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:48032)
0924 10:13:20.558758 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.558850 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 10:13:20.559232 28646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:35299: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
I20260924 10:13:20.660822 28720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } attempt: 4
W20260924 10:13:20.660722 28728 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:13:20.661118 28728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:20.661242 28728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:20.661408 28728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 10:13:20.662066 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.661580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:48044)
0924 10:13:20.661806 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.661904 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 10:13:20.663236 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.662799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:48054)
0924 10:13:20.662992 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.663119 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 10:13:20.663532 28720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:35299: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
I20260924 10:13:20.665446 28646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } attempt: 3
I20260924 10:13:20.666448 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.665966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:48058)
0924 10:13:20.666178 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.666271 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 10:13:20.667546 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.667169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:48064)
0924 10:13:20.667334 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.667427 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 10:13:20.667949 28646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:35299: Network error: Client connection negotiation failed: client connection to 127.6.2.252:35299: connect: Connection refused (error 111)
W20260924 10:13:20.670295 28728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:20.670377 28728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:20.670464 28728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:20.670552 28728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:20.670651 28728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:20.681185 28728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:42313,127.6.2.253:41159,127.6.2.252:35299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.252.keytab
--principal=kudu/127.6.2.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.252:35299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.2.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:20.682552 28728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:20.684473 28728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:20.698088 28736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:20.698148 28734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:20.698364 28733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:20.700822 28728 server_base.cc:1034] running on GCE node
I20260924 10:13:20.701823 28728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:20.703809 28728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:20.705078 28728 hybrid_clock.cc:648] HybridClock initialized: now 1790244800704977 us; error 56 us; skew 500 ppm
I20260924 10:13:20.705494 28728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:20 dist-test-slave-s3n3 krb5kdc[28566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244800, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:20.709594 28728 init.cc:377] Logged in from keytab as kudu/127.6.2.252@KRBTEST.COM (short username kudu)
I20260924 10:13:20.711459 28728 webserver.cc:460] Webserver started at http://127.6.2.252:46557/ using document root <none> and password file <none>
I20260924 10:13:20.712360 28728 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:20.712453 28728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:20.712709 28728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:20.715936 28728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/data/instance:
uuid: "499676c8adcd4fb39fcf9d51a3689751"
format_stamp: "Formatted at 2026-09-24 10:13:20 on dist-test-slave-s3n3"
server_key: "16d6f2eb1646142c299ecfc987deb4d4"
server_key_iv: "ae2eb9bb471bd13dd3c0386aa3f8f459"
server_key_version: "encryptionkey@0"
I20260924 10:13:20.716674 28728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/wal/instance:
uuid: "499676c8adcd4fb39fcf9d51a3689751"
format_stamp: "Formatted at 2026-09-24 10:13:20 on dist-test-slave-s3n3"
server_key: "16d6f2eb1646142c299ecfc987deb4d4"
server_key_iv: "ae2eb9bb471bd13dd3c0386aa3f8f459"
server_key_version: "encryptionkey@0"
I20260924 10:13:20.721824 28728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:13:20.725412 28743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:20.726871 28728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 10:13:20.727027 28728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/wal
uuid: "499676c8adcd4fb39fcf9d51a3689751"
format_stamp: "Formatted at 2026-09-24 10:13:20 on dist-test-slave-s3n3"
server_key: "16d6f2eb1646142c299ecfc987deb4d4"
server_key_iv: "ae2eb9bb471bd13dd3c0386aa3f8f459"
server_key_version: "encryptionkey@0"
I20260924 10:13:20.727301 28728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:20.741745 28728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:20.746229 28728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:20.746474 28728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:20.766301 28728 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.252:35299
I20260924 10:13:20.766309 28794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.252:35299 every 8 connection(s)
I20260924 10:13:20.767810 28728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/data/info.pb
I20260924 10:13:20.771615 28795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:20.773252 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 28728
I20260924 10:13:20.773458 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790244437048964-6155-0/minicluster-data/master-2/wal/instance
I20260924 10:13:20.773942 6155 external_mini_cluster.cc:1428] Setting key 3cfcd8c13c6c3e0603b4e5e3adf49efe
I20260924 10:13:20.781025 28795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:13:20 dist-test-slave-s3n3 krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.252@KRBTEST.COM
Sep 24 10:13:20 dist-test-slave-s3n3 krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244800, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.252@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:13:20.800217 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.783919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.6.2.254:42313)
0924 10:13:20.784587 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.784594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.785165 (+ 571us) server_negotiation.cc:407] Connection header received
0924 10:13:20.785389 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.785396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.785514 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.785656 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.786996 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.788034 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.789440 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.789844 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.790013 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.793691 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:20.793702 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:20.793709 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:20.793763 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:20.796957 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.797871 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.797876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.797878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.798029 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.798494 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.798502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.798508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.798854 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:20.798999 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:20.799578 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 10:13:20.799826 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":323,"threads_started":1}
I20260924 10:13:20.800567 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.775943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.6.2.252:35299)
0924 10:13:20.776804 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.776816 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.776879 (+ 63us) server_negotiation.cc:407] Connection header received
0924 10:13:20.778543 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.778566 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.778992 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.779505 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.781327 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.783167 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.785084 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.786247 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.786495 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.792132 (+ 5637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:20.792148 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:20.792167 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:20.792209 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:20.795697 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.796402 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.796411 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.796419 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.796572 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.796939 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.796946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.796949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.797679 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:20.797966 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:20.798752 (+ 786us) server_negotiation.cc:299] Negotiation successful
0924 10:13:20.799225 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":374,"threads_started":1}
I20260924 10:13:20.800587 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.783849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:42313 (local address 127.0.0.1:58116)
0924 10:13:20.784988 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.785044 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:20.785224 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:20.785755 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:20.785761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:20.786449 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:20.786788 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.786799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.788205 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.788211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.789276 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.789288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.790119 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.790125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.790257 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.791021 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:20.791046 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:20.793482 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:20.797122 (+ 3640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.797131 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.797138 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.797673 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.798158 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.798163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.798166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.798321 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.799100 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:20.799110 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:20.799289 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 10:13:20.799437 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 10:13:20.799657 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":719,"threads_started":1}
I20260924 10:13:20.802816 28795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:13:20 dist-test-slave-s3n3 krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244800, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.252@KRBTEST.COM for kudu/127.6.2.253@KRBTEST.COM
Sep 24 10:13:20 dist-test-slave-s3n3 krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:13:20.817260 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.803651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:41159 (local address 127.0.0.1:36258)
0924 10:13:20.803885 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.803903 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:20.804017 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:20.804375 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:20.804379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:20.804692 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:20.804822 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.804830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.806150 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.806156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.807288 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.807300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.808224 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.808229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.808422 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.809001 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:20.809024 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:20.811251 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:20.814741 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.814746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.814750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.815206 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.815598 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.815603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.815607 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.815760 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.816274 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:20.816279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:20.816434 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 10:13:20.816569 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 10:13:20.816869 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260924 10:13:20.817286 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.803708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.6.2.253:41159)
0924 10:13:20.803905 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.803910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.803960 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:13:20.804153 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.804157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.804220 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.804338 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.804982 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.805972 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.807429 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.808067 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.808207 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.811456 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:20.811466 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:20.811471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:20.811515 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:20.814581 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.815329 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.815334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.815338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.815478 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.815874 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.815878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.815882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.816158 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:20.816313 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:20.816668 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 10:13:20.816882 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:13:20.818315 28795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:13:20.819525 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.803641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.6.2.254:42313)
0924 10:13:20.803807 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.803813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.803873 (+ 60us) server_negotiation.cc:407] Connection header received
0924 10:13:20.804078 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.804086 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.804172 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.804254 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.805690 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.806595 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.808200 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.808868 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.808998 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.813792 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:20.813797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:20.813800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:20.813835 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:20.816586 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.817290 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.817302 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.817305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.817492 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.817933 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.817940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.817946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.818312 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:20.818614 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:20.818941 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 10:13:20.819194 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 10:13:20 dist-test-slave-s3n3 krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244800, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.252@KRBTEST.COM for kudu/127.6.2.252@KRBTEST.COM
Sep 24 10:13:20 dist-test-slave-s3n3 krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.253@KRBTEST.COM
I20260924 10:13:20.833159 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.819018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:48088)
0924 10:13:20.819220 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.819238 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:20.819337 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:20.819827 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:20.819831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:20.820337 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:20.820512 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.820527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.821776 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.821785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.823250 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.823261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.824152 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.824156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.824393 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.824950 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:20.824973 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:20.827439 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:20.830533 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.830539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.830543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.830916 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.831419 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.831448 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.831452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.831621 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.832294 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:20.832297 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:20.832417 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 10:13:20.832546 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 10:13:20.832758 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":17}
I20260924 10:13:20.833395 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.819277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.6.2.252:35299)
0924 10:13:20.819431 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.819437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.819451 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:13:20.819496 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.819499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.819557 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.819670 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.820658 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.821650 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.823369 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.824017 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.824139 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.827628 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:20.827633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:20.827636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:20.827662 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:20.830369 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.831143 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.831147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.831149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.831271 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.831758 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.831766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.831772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.832170 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:20.832325 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:20.832696 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 10:13:20.832956 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":51}
I20260924 10:13:20.837849 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.821247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36270 (local address 127.6.2.253:41159)
0924 10:13:20.821403 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.821417 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.821436 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:13:20.821490 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.821495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.821556 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.821651 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.823183 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.824250 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.825962 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.826352 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.826514 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.831107 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:20.831114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:20.831119 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:20.831160 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:20.834723 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.835693 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.835700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.835704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.835873 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.836346 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.836351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.836355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.836704 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:20.836973 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:20.837253 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 10:13:20.837489 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:13:20.839581 28795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751: Bootstrap starting.
I20260924 10:13:20.842818 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:13:20.843477 28795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:20.844796 28795 log.cc:824] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751: Log is configured to *not* fsync() on all Append() calls
I20260924 10:13:20.845178 28646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } attempt: 4
I20260924 10:13:20.848472 28795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751: No bootstrap required, opened a new log
I20260924 10:13:20.846553 28621 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ebc117e058604c1f8dfa46743680ef72"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58122
Sep 24 10:13:20 dist-test-slave-s3n3 krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for kudu/127.6.2.252@KRBTEST.COM
I20260924 10:13:20.859298 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.846102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48096 (local address 127.6.2.252:35299)
0924 10:13:20.846318 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.846326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.846347 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:13:20.846405 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.846408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.846467 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.846586 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.847483 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.848231 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.849642 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.850143 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.850288 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.853502 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:20.853507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:20.853510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:20.853543 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:20.856512 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.857184 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.857189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.857193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.857327 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.857729 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.857734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.857738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.858088 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:20.858267 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:20.858646 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 10:13:20.858863 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:13:20.859303 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.845842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:48096)
0924 10:13:20.846083 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.846103 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:20.846229 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:20.846611 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:20.846616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:20.847128 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:20.847334 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.847343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.848378 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.848383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.849484 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.849497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.850313 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.850318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.850522 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.851218 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:20.851235 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:20.853308 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:20.856665 (+ 3357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.856672 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.856676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.857055 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.857445 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:20.857451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:20.857456 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:20.857615 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:20.858212 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:20.858217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:20.858395 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 10:13:20.858550 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 10:13:20.858868 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 10:13:20.860353 28795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } }
I20260924 10:13:20.860843 28795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499676c8adcd4fb39fcf9d51a3689751, State: Initialized, Role: FOLLOWER
I20260924 10:13:20.861392 28795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } }
I20260924 10:13:20.862388 28802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } } }
I20260924 10:13:20.862815 28802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:20.863755 28795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:20.864722 28646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72: Bootstrap starting.
I20260924 10:13:20.868496 28646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:20.869730 28646 log.cc:824] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72: Log is configured to *not* fsync() on all Append() calls
I20260924 10:13:20.873291 28646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72: No bootstrap required, opened a new log
W20260924 10:13:20.878587 28814 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:13:20.878871 28814 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:13:20.881289 28646 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } }
I20260924 10:13:20.881731 28646 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc117e058604c1f8dfa46743680ef72, State: Initialized, Role: FOLLOWER
I20260924 10:13:20.882254 28646 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } }
I20260924 10:13:20.883632 28816 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } } }
I20260924 10:13:20.884431 28816 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:20.884754 28646 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:20.884819 28621 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 10:13:20.885038 28621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:20.874910 28728 master_runner.cc:186] Error getting master registration for 127.6.2.254:42313: OK, instance_id { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" instance_seqno: 1790244799696818 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:13:20.888199 28728 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:13:20.888875 28621 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } }
I20260924 10:13:20.890834 28621 leader_election.cc:288] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [CANDIDATE]: Term 1 election: Requested vote from peers f9e04f65cfe2425b8cef3b7293d5cb96 (127.6.2.253:41159), 499676c8adcd4fb39fcf9d51a3689751 (127.6.2.252:35299)
I20260924 10:13:20.892204 28770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ebc117e058604c1f8dfa46743680ef72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "499676c8adcd4fb39fcf9d51a3689751"
I20260924 10:13:20.893358 28770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:20.898681 28770 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebc117e058604c1f8dfa46743680ef72 in term 1.
I20260924 10:13:20.899793 28595 leader_election.cc:302] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499676c8adcd4fb39fcf9d51a3689751, ebc117e058604c1f8dfa46743680ef72; no voters:
I20260924 10:13:20.900450 28816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:13:20.901331 28816 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [term 1 LEADER]: Becoming Leader. State: Replica: ebc117e058604c1f8dfa46743680ef72, State: Running, Role: LEADER
I20260924 10:13:20.903236 28816 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } }
I20260924 10:13:20.908697 28825 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebc117e058604c1f8dfa46743680ef72. Latest consensus state: current_term: 1 leader_uuid: "ebc117e058604c1f8dfa46743680ef72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } } }
I20260924 10:13:20.909798 28825 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:20.923951 28827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260924 10:13:20.936375 28600 tablet.cc:2369] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:13:20.946099 28827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:13:20.962699 28720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } attempt: 5
I20260924 10:13:20.988732 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.940364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.6.2.253:41159)
0924 10:13:20.940553 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.940559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.961482 (+ 20923us) server_negotiation.cc:407] Connection header received
0924 10:13:20.965202 (+ 3720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.965211 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.965304 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.965388 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.969502 (+ 4114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.970841 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.972839 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.973549 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.973653 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.978779 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:20.978787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:20.978792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:20.978838 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:20.983303 (+ 4465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.984812 (+ 1509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.984817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.984821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.985085 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.985790 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.985795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.985798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.986353 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:20.986690 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:20.987629 (+ 939us) server_negotiation.cc:299] Negotiation successful
0924 10:13:20.988331 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:13:20.989841 28840 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:13:20.995221 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.934166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.6.2.254:42313)
0924 10:13:20.934350 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.934357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.957738 (+ 23381us) server_negotiation.cc:407] Connection header received
0924 10:13:20.961832 (+ 4094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.961840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.961939 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.962093 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.968385 (+ 6292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.969508 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.972134 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.972849 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.972947 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.982508 (+ 9561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:20.982516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:20.982521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:20.982585 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:20.986226 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.990459 (+ 4233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.990465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.990468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.990622 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:20.993309 (+ 2687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:20.993316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:20.993321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:20.993699 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:20.993966 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:20.994542 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 10:13:20.994801 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 10:13:20.998548 28749 tablet.cc:2369] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:13:20 dist-test-slave-s3n3 krb5kdc[28566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244800, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.253@KRBTEST.COM for kudu/127.6.2.252@KRBTEST.COM
I20260924 10:13:21.003360 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.958441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.6.2.252:35299)
0924 10:13:20.958644 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.958650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.964615 (+ 5965us) server_negotiation.cc:407] Connection header received
0924 10:13:20.964831 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.964838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.964924 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.965059 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.975851 (+ 10792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.976981 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.987186 (+ 10205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.987953 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.988073 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:20.996133 (+ 8060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:20.996139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:20.996145 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:20.996197 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:21.000089 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:21.000923 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:21.000929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:21.000933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:21.001103 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:21.001554 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:21.001559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:21.001564 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:21.001891 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:21.002150 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.002669 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.002909 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 10:13:20.994385 28579 master_runner.cc:186] Error getting master registration for 127.6.2.253:41159: OK, instance_id { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" instance_seqno: 1790244800234901 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:13:21.010528 28579 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:13:21.011479 28770 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751 [term 1 FOLLOWER]: Refusing update from remote peer ebc117e058604c1f8dfa46743680ef72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:13:21.012799 28825 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:13:21.026235 28802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebc117e058604c1f8dfa46743680ef72. Latest consensus state: current_term: 1 leader_uuid: "ebc117e058604c1f8dfa46743680ef72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } } }
I20260924 10:13:21.026569 28802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:21.028127 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.963317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:35299 (local address 127.0.0.1:48116)
0924 10:13:20.963696 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:20.963728 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:20.963843 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:20.987349 (+ 23506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:20.987355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:20.987900 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:20.988095 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.988107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.993355 (+ 5248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.993362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.994849 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:20.994865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.999203 (+ 4338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.999209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:20.999438 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.000142 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:21.000166 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:21.002972 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:21.024893 (+ 21921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:21.024902 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:21.024908 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:21.025573 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:21.026105 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:21.026109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:21.026113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:21.026295 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:21.027021 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:21.027026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:21.027208 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 10:13:21.027382 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 10:13:21.027715 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260924 10:13:21.029523 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:20.963669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.6.2.252:35299)
0924 10:13:20.983799 (+ 20130us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:20.983809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:20.983834 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:13:20.983893 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:20.983898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:20.983992 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:20.989507 (+ 5515us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:20.989791 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.991794 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:20.995096 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:20.997092 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.003191 (+ 6099us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.013706 (+ 10515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:21.013722 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:21.013730 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:21.013783 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:21.024677 (+ 10894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:21.025743 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:21.025748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:21.025751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:21.025944 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:21.026447 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:21.026453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:21.026459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:21.026875 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:21.028765 (+ 1890us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.028884 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.029109 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19933,"thread_start_us":23448,"threads_started":1}
I20260924 10:13:21.030520 28839 mvcc.cc:205] Tried to move back new op lower bound from 7332842704841949184 to 7332842704505266176. Current Snapshot: MvccSnapshot[applied={T|T < 7332842704841949184}]
I20260924 10:13:21.037676 28720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96: Bootstrap starting.
I20260924 10:13:21.042812 28720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:21.044570 28720 log.cc:824] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96: Log is configured to *not* fsync() on all Append() calls
I20260924 10:13:21.045830 28802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebc117e058604c1f8dfa46743680ef72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } } }
I20260924 10:13:21.046139 28802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:21.054019 28816 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebc117e058604c1f8dfa46743680ef72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } } }
I20260924 10:13:21.054363 28816 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:21.055125 28827 catalog_manager.cc:1347] Generated new cluster ID: aaa4f27e920c447a8f47a3a2ade595a7
I20260924 10:13:21.055226 28827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:13:21.077435 28720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96: No bootstrap required, opened a new log
I20260924 10:13:21.080353 28827 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:13:21.082237 28827 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:13:21.100570 28827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72: Generated new TSK 0
I20260924 10:13:21.101761 28827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:13:21.105173 28720 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } }
I20260924 10:13:21.105682 28720 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e04f65cfe2425b8cef3b7293d5cb96, State: Initialized, Role: FOLLOWER
I20260924 10:13:21.106256 28720 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } }
I20260924 10:13:21.107786 28847 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } } }
I20260924 10:13:21.108562 28847 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:21.109026 28720 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:21.109387 28694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:21.113863 28694 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96 [term 1 FOLLOWER]: Refusing update from remote peer ebc117e058604c1f8dfa46743680ef72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:13:21.114586 28695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ebc117e058604c1f8dfa46743680ef72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96"
I20260924 10:13:21.117908 28695 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebc117e058604c1f8dfa46743680ef72 in term 1.
I20260924 10:13:21.118674 28846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:13:21.189365 28857 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebc117e058604c1f8dfa46743680ef72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } } }
I20260924 10:13:21.190317 28847 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebc117e058604c1f8dfa46743680ef72. Latest consensus state: current_term: 1 leader_uuid: "ebc117e058604c1f8dfa46743680ef72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } } }
I20260924 10:13:21.190666 28847 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:21.190927 28857 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:21.214320 28864 catalog_manager.cc:1259] Loaded cluster ID: aaa4f27e920c447a8f47a3a2ade595a7
I20260924 10:13:21.214535 28864 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96: loading cluster ID for follower catalog manager: success
I20260924 10:13:21.218564 28864 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96: acquiring CA information for follower catalog manager: success
I20260924 10:13:21.221225 28864 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f9e04f65cfe2425b8cef3b7293d5cb96: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 10:13:21.237612 6155 mini_kdc.cc:348] Time spent kdestroy: real 0.115s user 0.000s sys 0.026s
W20260924 10:13:21.300083 28799 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:21.273868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.6.2.254:42313)
0924 10:13:21.274066 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.274074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.295509 (+ 21435us) server_negotiation.cc:407] Connection header received
0924 10:13:21.298413 (+ 2904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.298425 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.298532 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.298610 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:21.299822 (+ 1212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58136: BlockingRecv error: recv got EOF from 127.0.0.1:58136 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 10:13:21.303966 28725 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:21.280660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36294 (local address 127.6.2.253:41159)
0924 10:13:21.280868 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.280874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.301656 (+ 20782us) server_negotiation.cc:407] Connection header received
0924 10:13:21.302386 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.302393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.302511 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.302584 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:21.303724 (+ 1140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36294: BlockingRecv error: recv got EOF from 127.0.0.1:36294 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 10:13:21.306233 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:21.296148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.6.2.252:35299)
0924 10:13:21.296339 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.296345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.302123 (+ 5778us) server_negotiation.cc:407] Connection header received
0924 10:13:21.304902 (+ 2779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.304908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.305011 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.305085 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:21.305986 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48120: BlockingRecv error: recv got EOF from 127.0.0.1:48120 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:13:21.356141 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:21.337063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.6.2.253:41159)
0924 10:13:21.337261 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.337268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.347895 (+ 10627us) server_negotiation.cc:407] Connection header received
0924 10:13:21.348491 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.348498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.348753 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.348897 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:21.349599 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.350856 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.352926 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.353594 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.353735 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.354522 (+ 787us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:21.355280 (+ 758us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:21.355402 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.355656 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.355755 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 10:13:21.403035 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:21.389953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.6.2.252:35299)
0924 10:13:21.390147 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.390154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.400895 (+ 10741us) server_negotiation.cc:407] Connection header received
0924 10:13:21.401465 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.401474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.401606 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.401699 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:21.402750 (+ 1051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48134: BlockingRecv error: recv got EOF from 127.0.0.1:48134 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:13:21.454056 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:21.435945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.6.2.253:41159)
0924 10:13:21.436162 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.436169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.446827 (+ 10658us) server_negotiation.cc:407] Connection header received
0924 10:13:21.447375 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.447384 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.447496 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.447569 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:21.448284 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.449518 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.451678 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.452406 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.452524 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.453152 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:21.453310 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:21.453370 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.453620 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.453718 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 10:13:21.509140 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:21.495762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.6.2.252:35299)
0924 10:13:21.495978 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.495984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.507519 (+ 11535us) server_negotiation.cc:407] Connection header received
0924 10:13:21.507727 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.507734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.507851 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.507991 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:21.508899 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48150: BlockingRecv error: recv got EOF from 127.0.0.1:48150 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 10:13:21.521740 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:21.514706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.6.2.253:41159)
0924 10:13:21.514907 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.514914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.515471 (+ 557us) server_negotiation.cc:407] Connection header received
0924 10:13:21.515673 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.515680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.515782 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.515905 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:21.516319 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.517516 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.519511 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.520173 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.520315 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.520783 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:21.520964 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:21.521072 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.521259 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.521375 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 10:13:21.531769 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:21.529259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48166 (local address 127.6.2.252:35299)
0924 10:13:21.529476 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.529482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.530230 (+ 748us) server_negotiation.cc:407] Connection header received
0924 10:13:21.530405 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.530410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.530509 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.530641 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:21.531513 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48166: BlockingRecv error: recv got EOF from 127.0.0.1:48166 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:13:21.546922 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:21.539947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.6.2.253:41159)
0924 10:13:21.540151 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.540157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.540690 (+ 533us) server_negotiation.cc:407] Connection header received
0924 10:13:21.540903 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.540908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.541003 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.541102 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:21.541585 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.542714 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.544777 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.545426 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.545563 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.546148 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:21.546277 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:21.546367 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.546558 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.546648 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260924 10:13:21.556293 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:21.554260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48174 (local address 127.6.2.252:35299)
0924 10:13:21.554507 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.554513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.554594 (+ 81us) server_negotiation.cc:407] Connection header received
0924 10:13:21.554899 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.554905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.555013 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.555152 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:21.556043 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48174: BlockingRecv error: recv got EOF from 127.0.0.1:48174 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
I20260924 10:13:21.576180 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:21.568608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.6.2.253:41159)
0924 10:13:21.568809 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.568821 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.569380 (+ 559us) server_negotiation.cc:407] Connection header received
0924 10:13:21.569617 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.569623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.569724 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.569858 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:21.570325 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.571614 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.573715 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.574495 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.574660 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.575220 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:21.575385 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:21.575502 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.575703 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.575802 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 10:13:21.587538 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:21.585141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.6.2.252:35299)
0924 10:13:21.585356 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.585362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.585907 (+ 545us) server_negotiation.cc:407] Connection header received
0924 10:13:21.586126 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.586134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.586240 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.586365 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:21.587280 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48178: BlockingRecv error: recv got EOF from 127.0.0.1:48178 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 10:13:21.600323 28846 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebc117e058604c1f8dfa46743680ef72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc117e058604c1f8dfa46743680ef72" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 42313 } } peers { permanent_uuid: "f9e04f65cfe2425b8cef3b7293d5cb96" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 41159 } } peers { permanent_uuid: "499676c8adcd4fb39fcf9d51a3689751" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 35299 } } }
I20260924 10:13:21.600659 28846 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc117e058604c1f8dfa46743680ef72 [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:21.614845 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:21.607565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.6.2.253:41159)
0924 10:13:21.607764 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.607770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.608339 (+ 569us) server_negotiation.cc:407] Connection header received
0924 10:13:21.608576 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.608581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.608714 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.608818 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:21.609302 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.610565 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.612542 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.613252 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.613396 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.613969 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:21.614144 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:21.614262 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.614434 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.614536 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 10:13:21.633082 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:21.630075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.6.2.252:35299)
0924 10:13:21.630259 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.630265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.631451 (+ 1186us) server_negotiation.cc:407] Connection header received
0924 10:13:21.631687 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.631694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.631793 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.631914 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:21.632826 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48188: BlockingRecv error: recv got EOF from 127.0.0.1:48188 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:13:21.677206 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:21.670511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.6.2.253:41159)
0924 10:13:21.670788 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.670799 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.671429 (+ 630us) server_negotiation.cc:407] Connection header received
0924 10:13:21.671626 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.671638 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.671738 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.671867 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:21.672312 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.673533 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.674938 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.675706 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.675852 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.676266 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:21.676426 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:21.676539 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.676693 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.676792 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 10:13:21.685611 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:21.683498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.6.2.252:35299)
0924 10:13:21.683722 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.683728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.684080 (+ 352us) server_negotiation.cc:407] Connection header received
0924 10:13:21.684239 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.684245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.684388 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.684521 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:21.685342 (+ 821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48200: BlockingRecv error: recv got EOF from 127.0.0.1:48200 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 10:13:21.759490 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:21.753814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.6.2.253:41159)
0924 10:13:21.754186 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.754196 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.754758 (+ 562us) server_negotiation.cc:407] Connection header received
0924 10:13:21.754996 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.755001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.755140 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.755243 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:21.755662 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.756456 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.757773 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.758199 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.758348 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.758759 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:21.758865 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:21.758964 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.759133 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.759219 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260924 10:13:21.768535 28838 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:21.766136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.6.2.252:35299)
0924 10:13:21.766321 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.766325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.767005 (+ 680us) server_negotiation.cc:407] Connection header received
0924 10:13:21.767278 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.767283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.767371 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.767491 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:21.768281 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48204: BlockingRecv error: recv got EOF from 127.0.0.1:48204 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:13:21.881035 28814 catalog_manager.cc:1259] Loaded cluster ID: aaa4f27e920c447a8f47a3a2ade595a7
I20260924 10:13:21.881140 28814 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751: loading cluster ID for follower catalog manager: success
I20260924 10:13:21.883674 28814 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751: acquiring CA information for follower catalog manager: success
I20260924 10:13:21.885651 28814 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 499676c8adcd4fb39fcf9d51a3689751: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:13:21.914121 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:21.908682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36370 (local address 127.6.2.253:41159)
0924 10:13:21.909051 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.909058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.909327 (+ 269us) server_negotiation.cc:407] Connection header received
0924 10:13:21.909610 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.909615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.909704 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.909811 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:21.910254 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.911046 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.912402 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.912803 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.912956 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.913373 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:21.913489 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:21.913595 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.913751 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.913843 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 10:13:21.925686 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:21.920095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.6.2.252:35299)
0924 10:13:21.920323 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.920335 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.920469 (+ 134us) server_negotiation.cc:407] Connection header received
0924 10:13:21.920628 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.920631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.920785 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.920902 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:21.921419 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.922142 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.923519 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.923954 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.924056 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.924533 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:21.925083 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:21.925201 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.925329 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.925411 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 10:13:21.941076 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:21.932640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.6.2.253:41159)
0924 10:13:21.932868 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.932874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.934515 (+ 1641us) server_negotiation.cc:407] Connection header received
0924 10:13:21.934767 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.934776 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.934901 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.935025 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:21.935921 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.936881 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.938885 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.939493 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.939586 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.940303 (+ 717us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:21.940460 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:21.940554 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.940733 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.940816 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 10:13:21.941342 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:21.933601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.6.2.252:35299)
0924 10:13:21.933787 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.933793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.935220 (+ 1427us) server_negotiation.cc:407] Connection header received
0924 10:13:21.935432 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.935438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.935542 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.935634 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:21.936196 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.937184 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.939334 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.939956 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.940078 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.940550 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:21.940692 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:21.940776 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.940983 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.941085 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:13:21.942446 28971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:21.932444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.6.2.254:42313)
0924 10:13:21.935791 (+ 3347us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:21.935797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:21.935814 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:13:21.935932 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:21.935937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:21.936149 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:21.936335 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:21.936857 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.938018 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.939889 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:21.940526 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:21.940652 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:21.941293 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:21.941786 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:21.941918 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:21.942048 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 10:13:21.942129 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3211,"thread_start_us":3074,"threads_started":1}
I20260924 10:13:21.947852 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 28579
I20260924 10:13:21.949177 28641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:13:22.061471 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 28579
I20260924 10:13:22.083914 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 28653
I20260924 10:13:22.085587 28715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:13:22.205289 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 28653
I20260924 10:13:22.225914 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 28728
I20260924 10:13:22.227164 28790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:13:22.336838 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 28728
2026-09-24T10:13:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4456 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:13:22 dist-test-slave-s3n3 krb5kdc[28979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:13:22 dist-test-slave-s3n3 krb5kdc[28979](info): set up 2 sockets
Sep 24 10:13:22 dist-test-slave-s3n3 krb5kdc[28979](info): commencing operation
krb5kdc: starting...
W20260924 10:13:23.459712 6155 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/test-user.keytab.
Sep 24 10:13:23 dist-test-slave-s3n3 krb5kdc[28979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:13:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:13:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.2.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
Entry for principal kudu/127.6.2.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab.
I20260924 10:13:23.732394 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:40545
--webserver_interface=127.6.2.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:40545,127.6.2.253:45977,127.6.2.252:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:13:24.067104 28992 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:13:24.067528 28992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:24.067662 28992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:24.067811 28992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:24.075220 28992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:24.075371 28992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:24.075469 28992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:24.075572 28992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:24.075660 28992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:24.085906 28992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:40545,127.6.2.253:45977,127.6.2.252:40753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.254.keytab
--principal=kudu/127.6.2.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.254:40545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.2.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:24.087345 28992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:24.088701 28992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:24.101409 28997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:24.101454 28998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:24.101380 29000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:24.102685 28992 server_base.cc:1034] running on GCE node
I20260924 10:13:24.103596 28992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:24.105141 28992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:24.106446 28992 hybrid_clock.cc:648] HybridClock initialized: now 1790244804106372 us; error 40 us; skew 500 ppm
I20260924 10:13:24.106835 28992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:24 dist-test-slave-s3n3 krb5kdc[28979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:24.115181 28992 init.cc:377] Logged in from keytab as kudu/127.6.2.254@KRBTEST.COM (short username kudu)
I20260924 10:13:24.116823 28992 webserver.cc:460] Webserver started at http://127.6.2.254:40319/ using document root <none> and password file <none>
I20260924 10:13:24.117549 28992 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:24.117691 28992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:24.118027 28992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:24.121070 28992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/data/instance:
uuid: "5996c5d602674342be1a5b3f18e69d6f"
format_stamp: "Formatted at 2026-09-24 10:13:24 on dist-test-slave-s3n3"
server_key: "e7dbdc90808fb0d6e023c4f19afe5533"
server_key_iv: "a45fd873fbb1b5c61286231a32b2c8f4"
server_key_version: "encryptionkey@0"
I20260924 10:13:24.121855 28992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/wal/instance:
uuid: "5996c5d602674342be1a5b3f18e69d6f"
format_stamp: "Formatted at 2026-09-24 10:13:24 on dist-test-slave-s3n3"
server_key: "e7dbdc90808fb0d6e023c4f19afe5533"
server_key_iv: "a45fd873fbb1b5c61286231a32b2c8f4"
server_key_version: "encryptionkey@0"
I20260924 10:13:24.126973 28992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 10:13:24.130332 29007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:24.131671 28992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:13:24.131850 28992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/wal
uuid: "5996c5d602674342be1a5b3f18e69d6f"
format_stamp: "Formatted at 2026-09-24 10:13:24 on dist-test-slave-s3n3"
server_key: "e7dbdc90808fb0d6e023c4f19afe5533"
server_key_iv: "a45fd873fbb1b5c61286231a32b2c8f4"
server_key_version: "encryptionkey@0"
I20260924 10:13:24.132069 28992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:24.164014 28992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:24.168803 28992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:24.169135 28992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:24.190069 28992 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.254:40545
I20260924 10:13:24.190078 29058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.254:40545 every 8 connection(s)
I20260924 10:13:24.191483 28992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/data/info.pb
I20260924 10:13:24.195134 29059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:24.200229 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 28992
I20260924 10:13:24.200418 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-0/wal/instance
I20260924 10:13:24.200878 6155 external_mini_cluster.cc:1428] Setting key cdf1f6baaaa59afcca09eedbb0d47f19
I20260924 10:13:24.201763 29059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:13:24 dist-test-slave-s3n3 krb5kdc[28979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:13:24.226465 29061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.204987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.6.2.254:40545)
0924 10:13:24.205868 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:24.205885 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:24.206430 (+ 545us) server_negotiation.cc:407] Connection header received
0924 10:13:24.208529 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:24.208539 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:24.208846 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:24.209074 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:24.210426 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.211943 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.214164 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.214725 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.214856 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:24.219240 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:24.219248 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:24.219261 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:24.219292 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:24.222080 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:24.222962 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:24.222968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:24.222970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:24.223198 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:24.223685 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:24.223688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:24.223690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:24.224200 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:24.224361 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:24.225074 (+ 713us) server_negotiation.cc:299] Negotiation successful
0924 10:13:24.225405 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":644,"thread_start_us":419,"threads_started":1}
I20260924 10:13:24.226497 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.204987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:40545 (local address 127.0.0.1:47302)
0924 10:13:24.206246 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.206293 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:24.208156 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:24.209130 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:24.209135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:24.209681 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:24.210163 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:24.210192 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.212115 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.212123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:24.214001 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:24.214019 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.214971 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.214980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:24.215229 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:24.215932 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:24.215967 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:24.218943 (+ 2976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:24.222212 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:24.222226 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:24.222242 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:24.222838 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:24.223334 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:24.223342 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:24.223349 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:24.223571 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:24.224366 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:24.224377 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:24.224655 (+ 278us) client_negotiation.cc:769] Sending connection context
0924 10:13:24.224976 (+ 321us) client_negotiation.cc:240] Negotiation successful
0924 10:13:24.225601 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"mutex_wait_us":96,"thread_start_us":807,"threads_started":1}
WARNING: no policy specified for kudu/127.6.2.253@KRBTEST.COM; defaulting to no policy
I20260924 10:13:24.229512 29059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.253@KRBTEST.COM" created.
I20260924 10:13:24.230912 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.230434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:45977 (local address 127.0.0.1:43972)
0924 10:13:24.230647 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.230749 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 10:13:24.231201 29008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
I20260924 10:13:24.232784 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.232439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:45977 (local address 127.0.0.1:43974)
0924 10:13:24.232581 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.232665 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:13:24.233029 29059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:45977: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.253.keytab.
Entry for principal kudu/127.6.2.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.253.keytab.
I20260924 10:13:24.263516 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.253:45977
--webserver_interface=127.6.2.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.253.keytab
--principal=kudu/127.6.2.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:40545,127.6.2.253:45977,127.6.2.252:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:13:24.282428 29059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } attempt: 1
I20260924 10:13:24.283769 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.283296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:45977 (local address 127.0.0.1:43986)
0924 10:13:24.283504 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.283593 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 10:13:24.284917 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.284484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:45977 (local address 127.0.0.1:43996)
0924 10:13:24.284688 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.284766 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 10:13:24.285104 29059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:45977: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
I20260924 10:13:24.353370 29059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } attempt: 2
I20260924 10:13:24.354713 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.354241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:45977 (local address 127.0.0.1:43998)
0924 10:13:24.354476 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.354570 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 10:13:24.355856 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.355453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:45977 (local address 127.0.0.1:44014)
0924 10:13:24.355606 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.355723 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 10:13:24.356057 29059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:45977: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
I20260924 10:13:24.447325 29059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } attempt: 3
I20260924 10:13:24.448511 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.448062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:45977 (local address 127.0.0.1:44030)
0924 10:13:24.448272 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.448366 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 10:13:24.449692 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.449285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:45977 (local address 127.0.0.1:44044)
0924 10:13:24.449477 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.449548 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 10:13:24.449877 29059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:45977: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
I20260924 10:13:24.593132 29059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } attempt: 4
I20260924 10:13:24.594326 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.593798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:45977 (local address 127.0.0.1:44054)
0924 10:13:24.594113 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.594198 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260924 10:13:24.595420 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.595029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:45977 (local address 127.0.0.1:44058)
0924 10:13:24.595226 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.595304 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260924 10:13:24.595604 29059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.253:45977: Network error: Client connection negotiation failed: client connection to 127.6.2.253:45977: connect: Connection refused (error 111)
W20260924 10:13:24.598613 29066 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:13:24.599006 29066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:24.599164 29066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:24.599253 29066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:24.606684 29066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:24.606783 29066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:24.606840 29066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:24.606884 29066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:24.606938 29066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:24.617699 29066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:40545,127.6.2.253:45977,127.6.2.252:40753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.253.keytab
--principal=kudu/127.6.2.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.253:45977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.2.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:24.619024 29066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:24.620365 29066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:24.633265 29071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:24.633318 29072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:24.633289 29074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:24.633965 29066 server_base.cc:1034] running on GCE node
I20260924 10:13:24.634771 29066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:24.636492 29066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:24.637768 29066 hybrid_clock.cc:648] HybridClock initialized: now 1790244804637706 us; error 47 us; skew 500 ppm
I20260924 10:13:24.638125 29066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:24 dist-test-slave-s3n3 krb5kdc[28979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:24.641317 29066 init.cc:377] Logged in from keytab as kudu/127.6.2.253@KRBTEST.COM (short username kudu)
I20260924 10:13:24.642881 29066 webserver.cc:460] Webserver started at http://127.6.2.253:46375/ using document root <none> and password file <none>
I20260924 10:13:24.643669 29066 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:24.643802 29066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:24.644192 29066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:24.647308 29066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/data/instance:
uuid: "7d5d3f99e76a46708089ca2473a45962"
format_stamp: "Formatted at 2026-09-24 10:13:24 on dist-test-slave-s3n3"
server_key: "a2b3064239a25aa637ee9902e621edb8"
server_key_iv: "7158dc52c0407962562a3c1c99f00d67"
server_key_version: "encryptionkey@0"
I20260924 10:13:24.648123 29066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/wal/instance:
uuid: "7d5d3f99e76a46708089ca2473a45962"
format_stamp: "Formatted at 2026-09-24 10:13:24 on dist-test-slave-s3n3"
server_key: "a2b3064239a25aa637ee9902e621edb8"
server_key_iv: "7158dc52c0407962562a3c1c99f00d67"
server_key_version: "encryptionkey@0"
I20260924 10:13:24.653337 29066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 10:13:24.656885 29081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:24.658221 29066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:13:24.658413 29066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/wal
uuid: "7d5d3f99e76a46708089ca2473a45962"
format_stamp: "Formatted at 2026-09-24 10:13:24 on dist-test-slave-s3n3"
server_key: "a2b3064239a25aa637ee9902e621edb8"
server_key_iv: "7158dc52c0407962562a3c1c99f00d67"
server_key_version: "encryptionkey@0"
I20260924 10:13:24.658648 29066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:24.693635 29066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:24.698175 29066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:24.698477 29066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:24.718107 29066 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.253:45977
I20260924 10:13:24.718114 29132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.253:45977 every 8 connection(s)
I20260924 10:13:24.719570 29066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/data/info.pb
I20260924 10:13:24.721602 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 29066
I20260924 10:13:24.721822 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-1/wal/instance
I20260924 10:13:24.722301 6155 external_mini_cluster.cc:1428] Setting key 88992c681388708c1dc4b328cc0bc792
I20260924 10:13:24.723572 29133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:24.730278 29133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:13:24 dist-test-slave-s3n3 krb5kdc[28979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.253@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.2.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.2.252@KRBTEST.COM" created.
I20260924 10:13:24.752991 29136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.733148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.6.2.254:40545)
0924 10:13:24.733878 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:24.733884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:24.734591 (+ 707us) server_negotiation.cc:407] Connection header received
0924 10:13:24.736930 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:24.736947 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:24.737051 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:24.737188 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:24.738928 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.739804 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.741528 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.741985 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.742132 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:24.746326 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:24.746362 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:24.746366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:24.746415 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:24.749531 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:24.750358 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:24.750364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:24.750368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:24.750505 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:24.750985 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:24.750991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:24.750994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:24.751363 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:24.751556 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:24.752315 (+ 759us) server_negotiation.cc:299] Negotiation successful
0924 10:13:24.752563 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":334,"threads_started":1}
I20260924 10:13:24.754055 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.733586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:40545 (local address 127.0.0.1:47312)
0924 10:13:24.734391 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.734445 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:24.736598 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:24.737457 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:24.737466 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:24.738072 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:24.738684 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:24.738703 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.739997 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.740003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:24.741337 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:24.741352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.742240 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.742248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:24.742519 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:24.743300 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:24.743326 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:24.746062 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:24.749700 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:24.749710 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:24.749735 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:24.750217 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:24.750635 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:24.750640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:24.750645 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:24.750853 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:24.751521 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:24.751530 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:24.751920 (+ 390us) client_negotiation.cc:769] Sending connection context
0924 10:13:24.752220 (+ 300us) client_negotiation.cc:240] Negotiation successful
0924 10:13:24.752823 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":346,"threads_started":1}
I20260924 10:13:24.756112 29133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:13:24 dist-test-slave-s3n3 krb5kdc[28979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.253@KRBTEST.COM for kudu/127.6.2.253@KRBTEST.COM
I20260924 10:13:24.771605 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.756977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:45977 (local address 127.0.0.1:44064)
0924 10:13:24.757162 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.757183 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:24.757287 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:24.758376 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:24.758380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:24.758832 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:24.758971 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:24.758980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.760555 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.760563 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:24.761444 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:24.761463 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.762325 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.762331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:24.762637 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:24.763301 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:24.763319 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:24.765649 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:24.769054 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:24.769060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:24.769064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:24.769442 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:24.769819 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:24.769827 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:24.769832 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:24.769969 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:24.770561 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:24.770569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:24.770707 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 10:13:24.770862 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 10:13:24.771177 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":177}
I20260924 10:13:24.771648 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.757255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.6.2.253:45977)
0924 10:13:24.757959 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:24.757965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:24.757988 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:13:24.758033 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:24.758039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:24.758223 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:24.758390 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:24.759233 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.760383 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.761588 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.762184 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.762316 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:24.765871 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:24.765883 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:24.765896 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:24.765925 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:24.768849 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:24.769584 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:24.769593 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:24.769602 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:24.769713 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:24.770079 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:24.770084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:24.770087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:24.770439 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:24.770651 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:24.771005 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 10:13:24.771283 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":547,"thread_start_us":419,"threads_started":1}
I20260924 10:13:24.774374 29133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:13:24.776063 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.775135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48858)
0924 10:13:24.775532 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.775738 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260924 10:13:24.776468 29082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
I20260924 10:13:24.778330 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.777830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48866)
0924 10:13:24.777999 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.778131 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 10:13:24.778578 29133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:40753: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.2.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.252.keytab.
Entry for principal kudu/127.6.2.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.252.keytab.
I20260924 10:13:24.783723 6155 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
/tmp/dist-test-taskq3UYkq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.252:40753
--webserver_interface=127.6.2.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.252.keytab
--principal=kudu/127.6.2.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.2.212:43063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.2.254:40545,127.6.2.253:45977,127.6.2.252:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:13:24.827867 29133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } attempt: 1
I20260924 10:13:24.829128 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.828710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48878)
0924 10:13:24.828893 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.828983 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 10:13:24.830222 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.829813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48882)
0924 10:13:24.829962 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.830049 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 10:13:24.830616 29133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:40753: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
I20260924 10:13:24.894865 29059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } attempt: 5
I20260924 10:13:24.898860 29133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } attempt: 2
I20260924 10:13:24.899971 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.899475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48888)
0924 10:13:24.899670 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.899792 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 10:13:24.901278 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.900762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48896)
0924 10:13:24.900963 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.901071 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 10:13:24.901571 29133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:40753: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Sep 24 10:13:24 dist-test-slave-s3n3 krb5kdc[28979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for kudu/127.6.2.253@KRBTEST.COM
I20260924 10:13:24.910640 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.895963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.6.2.253:45977)
0924 10:13:24.896177 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:24.896184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:24.896208 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:13:24.896276 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:24.896281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:24.896389 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:24.896501 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:24.897325 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.898507 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.900365 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.901013 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.901165 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:24.904931 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:24.904937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:24.904948 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:24.904985 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:24.907853 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:24.908583 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:24.908591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:24.908595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:24.908730 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:24.909144 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:24.909157 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:24.909159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:24.909454 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:24.909620 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:24.910036 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 10:13:24.910271 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 10:13:24.910665 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.895722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:45977 (local address 127.0.0.1:44080)
0924 10:13:24.896017 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.896042 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:24.896176 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:24.896552 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:24.896555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:24.896960 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:24.897148 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:24.897157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.898725 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.898730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:24.900149 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:24.900167 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:24.901176 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:24.901182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:24.901474 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:24.902241 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:24.902260 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:24.904697 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:24.908021 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:24.908027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:24.908032 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:24.908439 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:24.908853 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:24.908858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:24.908862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:24.909014 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:24.909579 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:24.909585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:24.909752 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 10:13:24.909911 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 10:13:24.910269 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 10:13:24.911875 29059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:13:24.912925 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.912538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48902)
0924 10:13:24.912698 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.912780 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 10:13:24.913890 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.913520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48916)
0924 10:13:24.913672 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.913759 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 10:13:24.914172 29059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:40753: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
I20260924 10:13:24.965418 29059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } attempt: 1
I20260924 10:13:24.966668 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.966162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48930)
0924 10:13:24.966374 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.966476 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 10:13:24.967746 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.967400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48932)
0924 10:13:24.967555 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.967629 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 10:13:24.967967 29059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:40753: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
I20260924 10:13:24.992825 29133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } attempt: 3
I20260924 10:13:24.994138 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.993650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48936)
0924 10:13:24.993850 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.993939 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 10:13:24.995601 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:24.995113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48948)
0924 10:13:24.995351 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:24.995434 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260924 10:13:24.995893 29133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:40753: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
I20260924 10:13:25.036197 29059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } attempt: 2
I20260924 10:13:25.037415 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.036924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48960)
0924 10:13:25.037175 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:25.037260 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 10:13:25.038372 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.038037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48976)
0924 10:13:25.038180 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:25.038253 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:13:25.038568 29059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:40753: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
W20260924 10:13:25.128144 29141 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:13:25.128545 29141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:13:25.128674 29141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:13:25.128813 29141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:13:25.136478 29141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:13:25.136629 29141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:13:25.136740 29141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:13:25.136828 29141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:13:25.136950 29141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:13:25.139168 29133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } attempt: 4
I20260924 10:13:25.140321 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.139872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48982)
0924 10:13:25.140086 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:25.140180 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 10:13:25.141345 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.141034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:48998)
0924 10:13:25.141185 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:25.141256 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 10:13:25.141548 29133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:40753: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
I20260924 10:13:25.144992 29059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } attempt: 3
I20260924 10:13:25.146207 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.145768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:49000)
0924 10:13:25.145956 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:25.146041 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 10:13:25.147244 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.146826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:49016)
0924 10:13:25.147011 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:25.147127 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 10:13:25.147455 29059 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.2.252:40753: Network error: Client connection negotiation failed: client connection to 127.6.2.252:40753: connect: Connection refused (error 111)
I20260924 10:13:25.148077 29141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.2.212:43063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.2.254:40545,127.6.2.253:45977,127.6.2.252:40753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/krb5kdc/kudu_127.6.2.252.keytab
--principal=kudu/127.6.2.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.2.252:40753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.2.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 07:48:44 UTC on ccycloud.quasar-uwxnll.root.comops.site
build id 4669409
ASAN enabled
I20260924 10:13:25.149451 29141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:13:25.150772 29141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:13:25.162892 29146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:25.162966 29147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:13:25.162990 29149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:13:25.164081 29141 server_base.cc:1034] running on GCE node
I20260924 10:13:25.164867 29141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:13:25.166333 29141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:13:25.167559 29141 hybrid_clock.cc:648] HybridClock initialized: now 1790244805167496 us; error 32 us; skew 500 ppm
I20260924 10:13:25.167987 29141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 10:13:25 dist-test-slave-s3n3 krb5kdc[28979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244805, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:13:25.171278 29141 init.cc:377] Logged in from keytab as kudu/127.6.2.252@KRBTEST.COM (short username kudu)
I20260924 10:13:25.172848 29141 webserver.cc:460] Webserver started at http://127.6.2.252:42425/ using document root <none> and password file <none>
I20260924 10:13:25.173520 29141 fs_manager.cc:359] Metadata directory not provided
I20260924 10:13:25.173609 29141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:13:25.173862 29141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:13:25.176883 29141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/data/instance:
uuid: "7b66f74555c5460792ef9032a2e38e87"
format_stamp: "Formatted at 2026-09-24 10:13:25 on dist-test-slave-s3n3"
server_key: "bdb4bbc6ce30baabf142797d574f6ead"
server_key_iv: "7da57d72fa7920f45d3ee89bf47048d5"
server_key_version: "encryptionkey@0"
I20260924 10:13:25.177688 29141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/wal/instance:
uuid: "7b66f74555c5460792ef9032a2e38e87"
format_stamp: "Formatted at 2026-09-24 10:13:25 on dist-test-slave-s3n3"
server_key: "bdb4bbc6ce30baabf142797d574f6ead"
server_key_iv: "7da57d72fa7920f45d3ee89bf47048d5"
server_key_version: "encryptionkey@0"
I20260924 10:13:25.182864 29141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260924 10:13:25.186299 29156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:13:25.187747 29141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 10:13:25.187937 29141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/data,/tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/wal
uuid: "7b66f74555c5460792ef9032a2e38e87"
format_stamp: "Formatted at 2026-09-24 10:13:25 on dist-test-slave-s3n3"
server_key: "bdb4bbc6ce30baabf142797d574f6ead"
server_key_iv: "7da57d72fa7920f45d3ee89bf47048d5"
server_key_version: "encryptionkey@0"
I20260924 10:13:25.188202 29141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:13:25.222247 29141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:13:25.226773 29141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:13:25.227116 29141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:13:25.245994 29141 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.252:40753
I20260924 10:13:25.246018 29207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.2.252:40753 every 8 connection(s)
I20260924 10:13:25.247454 29141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/data/info.pb
I20260924 10:13:25.250407 6155 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu as pid 29141
I20260924 10:13:25.250615 6155 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskq3UYkq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790244437048964-6155-0/minicluster-data/master-2/wal/instance
I20260924 10:13:25.251237 6155 external_mini_cluster.cc:1428] Setting key 979e91ece41a9081db6853577d654487
I20260924 10:13:25.251506 29208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:13:25.261509 29208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:13:25 dist-test-slave-s3n3 krb5kdc[28979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.252@KRBTEST.COM
Sep 24 10:13:25 dist-test-slave-s3n3 krb5kdc[28979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244805, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.252@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
I20260924 10:13:25.277267 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.253040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.6.2.252:40753)
0924 10:13:25.253894 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.253905 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.253944 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:13:25.256406 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.256430 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.256895 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.257255 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.259223 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.260539 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.262353 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.263301 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.263469 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.268679 (+ 5210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:25.268686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:25.268696 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:25.268731 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:25.272597 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.273297 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.273309 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.273323 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.273510 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.273914 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.273917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.273919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.274486 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:25.274725 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:25.275312 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 10:13:25.275749 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":425,"threads_started":1}
I20260924 10:13:25.282091 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.264709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.254:40545 (local address 127.0.0.1:47320)
0924 10:13:25.265688 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:25.265746 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:25.265928 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:25.266313 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:25.266322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:25.267193 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:25.267473 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:25.267486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.268861 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.268869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:25.270637 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:25.270649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.271490 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.271496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:25.271659 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.272244 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:25.272269 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:25.274774 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:25.278715 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:25.278724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:25.278731 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:25.279426 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:25.279885 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:25.279890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:25.279894 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:25.280071 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:25.280901 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:25.280907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:25.281114 (+ 207us) client_negotiation.cc:769] Sending connection context
0924 10:13:25.281333 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 10:13:25.281661 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":534,"threads_started":1}
I20260924 10:13:25.282099 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.264716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.6.2.254:40545)
0924 10:13:25.265326 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.265332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.265853 (+ 521us) server_negotiation.cc:407] Connection header received
0924 10:13:25.266054 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.266058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.266121 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.266210 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.267641 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.268707 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.270819 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.271304 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.271400 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.274982 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:25.274990 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:25.274994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:25.275039 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:25.278544 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.279599 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.279607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.279612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.279763 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.280207 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.280212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.280215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.280638 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:25.280804 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:25.281451 (+ 647us) server_negotiation.cc:299] Negotiation successful
0924 10:13:25.281664 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":246,"threads_started":1}
I20260924 10:13:25.288726 29208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:13:25 dist-test-slave-s3n3 krb5kdc[28979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.254@KRBTEST.COM
Sep 24 10:13:25 dist-test-slave-s3n3 krb5kdc[28979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244805, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.252@KRBTEST.COM for kudu/127.6.2.253@KRBTEST.COM
I20260924 10:13:25.297214 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.281150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.6.2.254:40545)
0924 10:13:25.282316 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.282320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.282334 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:13:25.282378 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.282381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.282440 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.282554 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.283747 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.284469 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.285720 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.286170 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.286307 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.291410 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:25.291415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:25.291418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:25.291450 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:25.294351 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.295201 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.295206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.295208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.295333 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.295722 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.295725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.295727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.296030 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:25.296325 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:25.296588 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 10:13:25.296839 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063,"thread_start_us":6104,"threads_started":1}
I20260924 10:13:25.303745 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.289600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.253:45977 (local address 127.0.0.1:44090)
0924 10:13:25.289799 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:25.289820 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:25.289952 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:25.290356 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:25.290390 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:25.290973 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:25.291216 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:25.291228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.292333 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.292339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:25.293324 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:25.293337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.294045 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.294051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:25.294250 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.294750 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:25.294775 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:25.297321 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:25.301317 (+ 3996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:25.301321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:25.301324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:25.301675 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:25.302091 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:25.302096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:25.302100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:25.302244 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:25.302762 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:25.302767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:25.302926 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 10:13:25.303070 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 10:13:25.303378 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 10:13:25.303819 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.289630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.6.2.253:45977)
0924 10:13:25.289826 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.289830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.289879 (+ 49us) server_negotiation.cc:407] Connection header received
0924 10:13:25.290068 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.290072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.290183 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.290291 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.291393 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.292184 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.293473 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.293885 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.294024 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.297510 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:25.297515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:25.297518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:25.297552 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:25.301155 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.301797 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.301805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.301808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.301967 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.302365 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.302370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.302373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.302644 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:25.302801 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:25.303199 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 10:13:25.303424 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:13:25.304669 29208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:13:25 dist-test-slave-s3n3 krb5kdc[28979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.2.253@KRBTEST.COM
Sep 24 10:13:25 dist-test-slave-s3n3 krb5kdc[28979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244805, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.252@KRBTEST.COM for kudu/127.6.2.252@KRBTEST.COM
I20260924 10:13:25.315596 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.298873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.6.2.253:45977)
0924 10:13:25.299471 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.299475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.299493 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:13:25.299538 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.299541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.299597 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.299683 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.301252 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.302026 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.303272 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.303731 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.304358 (+ 627us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.309272 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:25.309277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:25.309280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:25.309311 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:25.312478 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.313221 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.313227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.313231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.313382 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.313881 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.313888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.313892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.314270 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:25.314562 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:25.314867 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 10:13:25.315154 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":314,"threads_started":1}
I20260924 10:13:25.318883 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.305361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:49026)
0924 10:13:25.305547 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:25.305567 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:25.305668 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:25.305963 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:25.305970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:25.306461 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:25.306699 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:25.306713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.307717 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.307725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:25.308682 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:25.308694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.309562 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.309568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:25.309796 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.310327 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:25.310353 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:25.312717 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:25.316246 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:25.316250 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:25.316252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:25.316665 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:25.317042 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:25.317047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:25.317050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:25.317204 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:25.317886 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:25.317890 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:25.318124 (+ 234us) client_negotiation.cc:769] Sending connection context
0924 10:13:25.318260 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 10:13:25.318579 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 10:13:25.319214 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.305470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.6.2.252:40753)
0924 10:13:25.305606 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.305610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.305624 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:13:25.305775 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.305779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.305831 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.305905 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.306844 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.307581 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.308825 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.309398 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.309492 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.312884 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:25.312889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:25.312892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:25.312919 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:25.316085 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.316784 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.316791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.316795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.316923 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.317319 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.317323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.317325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.317726 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:25.317895 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:25.318408 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 10:13:25.318780 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":46}
I20260924 10:13:25.320899 6155 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:13:25.324677 29059 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } attempt: 4
I20260924 10:13:25.325002 29208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87: Bootstrap starting.
I20260924 10:13:25.323961 29034 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5996c5d602674342be1a5b3f18e69d6f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47324
I20260924 10:13:25.329074 29208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:25.330374 29208 log.cc:824] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:13:25 dist-test-slave-s3n3 krb5kdc[28979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.254@KRBTEST.COM for kudu/127.6.2.252@KRBTEST.COM
I20260924 10:13:25.333968 29208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87: No bootstrap required, opened a new log
I20260924 10:13:25.341387 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.325350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:49040)
0924 10:13:25.325561 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:25.325586 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:25.325711 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:25.326153 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:25.326158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:25.326697 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:25.326966 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:25.326984 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.328434 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.328439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:25.329864 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:25.329880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.330891 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.330900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:25.331181 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.331924 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:25.331952 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:25.334785 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:25.338496 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:25.338502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:25.338506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:25.339013 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:25.339464 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:25.339470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:25.339473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:25.339657 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:25.340332 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:25.340338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:25.340499 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 10:13:25.340661 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 10:13:25.340976 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 10:13:25.341437 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.325620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.6.2.252:40753)
0924 10:13:25.325830 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.325836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.325854 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:13:25.325909 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.325914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.325997 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.326097 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.327153 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.328275 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.330015 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.330719 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.330851 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.335098 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:25.335106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:25.335113 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:25.335152 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:25.338368 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.339177 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.339183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.339186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.339340 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.339784 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.339789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.339796 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.340192 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:25.340352 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:25.340812 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 10:13:25.341052 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:13:25.345918 29208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } }
I20260924 10:13:25.346377 29208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b66f74555c5460792ef9032a2e38e87, State: Initialized, Role: FOLLOWER
I20260924 10:13:25.346886 29208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } }
I20260924 10:13:25.347963 29217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } } }
I20260924 10:13:25.348381 29217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:25.348387 29059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f: Bootstrap starting.
I20260924 10:13:25.349269 29208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:25.352774 29059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:25.354321 29059 log.cc:824] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:13:25.357956 29059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f: No bootstrap required, opened a new log
W20260924 10:13:25.362103 29228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:13:25.362279 29228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:13:25.365738 29059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } }
I20260924 10:13:25.366137 29059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5996c5d602674342be1a5b3f18e69d6f, State: Initialized, Role: FOLLOWER
I20260924 10:13:25.366616 29059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } }
I20260924 10:13:25.367812 29231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } } }
I20260924 10:13:25.368465 29231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:25.357293 29141 master_runner.cc:186] Error getting master registration for 127.6.2.254:40545: OK, instance_id { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" instance_seqno: 1790244804184808 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:13:25.368916 29141 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:13:25.368959 29059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:25.369072 29034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 10:13:25.369210 29034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:25.371897 29034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } }
I20260924 10:13:25.373019 29034 leader_election.cc:288] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [CANDIDATE]: Term 1 election: Requested vote from peers 7d5d3f99e76a46708089ca2473a45962 (127.6.2.253:45977), 7b66f74555c5460792ef9032a2e38e87 (127.6.2.252:40753)
I20260924 10:13:25.374092 29183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5996c5d602674342be1a5b3f18e69d6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7b66f74555c5460792ef9032a2e38e87"
I20260924 10:13:25.374552 29183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:25.378645 29183 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5996c5d602674342be1a5b3f18e69d6f in term 1.
I20260924 10:13:25.379369 29008 leader_election.cc:302] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5996c5d602674342be1a5b3f18e69d6f, 7b66f74555c5460792ef9032a2e38e87; no voters:
I20260924 10:13:25.379803 29231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:13:25.386921 29231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [term 1 LEADER]: Becoming Leader. State: Replica: 5996c5d602674342be1a5b3f18e69d6f, State: Running, Role: LEADER
I20260924 10:13:25.387742 29231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } }
I20260924 10:13:25.394004 29238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5996c5d602674342be1a5b3f18e69d6f. Latest consensus state: current_term: 1 leader_uuid: "5996c5d602674342be1a5b3f18e69d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } } }
I20260924 10:13:25.394644 29238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:25.401679 29244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:13:25.417493 29244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260924 10:13:25.422827 29013 tablet.cc:2369] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:13:25.440707 29133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } attempt: 5
I20260924 10:13:25.430539 28992 master_runner.cc:186] Error getting master registration for 127.6.2.253:45977: OK, instance_id { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" instance_seqno: 1790244804712550 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:13:25.448338 28992 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:13:25.461814 29183 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87 [term 1 FOLLOWER]: Refusing update from remote peer 5996c5d602674342be1a5b3f18e69d6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:13:25.463002 29238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:13:25.479297 29162 tablet.cc:2369] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:13:25 dist-test-slave-s3n3 krb5kdc[28979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790244804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.2.253@KRBTEST.COM for kudu/127.6.2.252@KRBTEST.COM
I20260924 10:13:25.484602 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.409229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.6.2.254:40545)
0924 10:13:25.409456 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.409466 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.427323 (+ 17857us) server_negotiation.cc:407] Connection header received
0924 10:13:25.430209 (+ 2886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.430217 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.430315 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.430402 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.437668 (+ 7266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.438877 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.457867 (+ 18990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.458620 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.458757 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.464936 (+ 6179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:25.464945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:25.464950 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:25.464995 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:25.468672 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.476239 (+ 7567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.476247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.476251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.476415 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.477901 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.477906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.477912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.478352 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:25.478642 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:25.483735 (+ 5093us) server_negotiation.cc:299] Negotiation successful
0924 10:13:25.484096 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:13:25.494577 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.411506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44106 (local address 127.6.2.253:45977)
0924 10:13:25.411785 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.411795 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.436120 (+ 24325us) server_negotiation.cc:407] Connection header received
0924 10:13:25.440886 (+ 4766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.440893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.440988 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.441063 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.452187 (+ 11124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.453290 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.468783 (+ 15493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.469616 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.469771 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.485464 (+ 15693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:25.485471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:25.485476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:25.485517 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:25.489187 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.491947 (+ 2760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.491953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.491957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.492111 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.492601 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.492607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.492611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.493052 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:25.493313 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:25.493970 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 10:13:25.494220 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:13:25.499782 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.427928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.6.2.252:40753)
0924 10:13:25.428110 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.428116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.452825 (+ 24709us) server_negotiation.cc:407] Connection header received
0924 10:13:25.453911 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.453919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.454019 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.454132 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.456098 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.457914 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.474029 (+ 16115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.474742 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.474870 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.490861 (+ 15991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:25.490868 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:25.490873 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:25.490918 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:25.495017 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.496301 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.496308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.496312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.496478 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.496942 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.496947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.496951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.497772 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:25.498069 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:25.499117 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0924 10:13:25.499355 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:13:25.502215 29217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5996c5d602674342be1a5b3f18e69d6f. Latest consensus state: current_term: 1 leader_uuid: "5996c5d602674342be1a5b3f18e69d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } } }
I20260924 10:13:25.502524 29217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:25.504267 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.441306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.2.252:40753 (local address 127.0.0.1:49056)
0924 10:13:25.441549 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 10:13:25.441581 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 10:13:25.441678 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:13:25.464942 (+ 23264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:13:25.464948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:13:25.465462 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:13:25.465678 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:25.465692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.470801 (+ 5109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.470807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:25.472274 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:13:25.472292 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.477980 (+ 5688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.477989 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:13:25.478220 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.478893 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:13:25.478918 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:13:25.482175 (+ 3257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:13:25.501038 (+ 18863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:25.501043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:25.501047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:25.501612 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:25.502302 (+ 690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:13:25.502309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:13:25.502313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:13:25.502483 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:13:25.503288 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:13:25.503292 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:13:25.503462 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 10:13:25.503617 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 10:13:25.503904 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260924 10:13:25.504691 29253 mvcc.cc:205] Tried to move back new op lower bound from 7332842723075985408 to 7332842722875789312. Current Snapshot: MvccSnapshot[applied={T|T < 7332842723075985408}]
I20260924 10:13:25.505561 29255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:25.441415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.6.2.252:40753)
0924 10:13:25.460252 (+ 18837us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.460258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.460280 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:13:25.460329 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.460334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.460432 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.465841 (+ 5409us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.466151 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.470655 (+ 4504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.474123 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:25.477351 (+ 3228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:25.482378 (+ 5027us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:25.482662 (+ 284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:13:25.482668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:13:25.482673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:13:25.482706 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:13:25.500881 (+ 18175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.501782 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.501817 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.501821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.502101 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:13:25.502677 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:13:25.502684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:13:25.502686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:13:25.503146 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:13:25.504719 (+ 1573us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:25.504875 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 10:13:25.505101 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18684,"thread_start_us":19108,"threads_started":1}
I20260924 10:13:25.505789 29256 mvcc.cc:205] Tried to move back new op lower bound from 7332842723075985408 to 7332842722875789312. Current Snapshot: MvccSnapshot[applied={T|T < 7332842723075985408}]
I20260924 10:13:25.506220 29231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5996c5d602674342be1a5b3f18e69d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } } }
I20260924 10:13:25.506542 29231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [sys.catalog]: This master's current role is: LEADER
I20260924 10:13:25.509330 29217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5996c5d602674342be1a5b3f18e69d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } } }
I20260924 10:13:25.509583 29217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:25.511363 29244 catalog_manager.cc:1347] Generated new cluster ID: ac68dcedfcb24dc3953129d393245b1d
I20260924 10:13:25.511454 29244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:13:25.515687 29133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962: Bootstrap starting.
I20260924 10:13:25.519940 29133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962: Neither blocks nor log segments found. Creating new log.
I20260924 10:13:25.522846 29133 log.cc:824] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962: Log is configured to *not* fsync() on all Append() calls
I20260924 10:13:25.527640 29244 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:13:25.529455 29244 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:13:25.532656 29133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962: No bootstrap required, opened a new log
I20260924 10:13:25.545549 29133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } }
I20260924 10:13:25.546512 29133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5d3f99e76a46708089ca2473a45962, State: Initialized, Role: FOLLOWER
I20260924 10:13:25.547127 29133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } }
I20260924 10:13:25.547303 29244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f: Generated new TSK 0
I20260924 10:13:25.548424 29244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:13:25.554129 29261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } } }
I20260924 10:13:25.554543 29261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:25.556918 29107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:13:25.556555 29108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5996c5d602674342be1a5b3f18e69d6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7d5d3f99e76a46708089ca2473a45962"
I20260924 10:13:25.558679 29133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:13:25.560899 29107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962 [term 1 FOLLOWER]: Refusing update from remote peer 5996c5d602674342be1a5b3f18e69d6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:13:25.564672 29108 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5996c5d602674342be1a5b3f18e69d6f in term 1.
I20260924 10:13:25.567247 29231 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:13:25.626158 29261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5996c5d602674342be1a5b3f18e69d6f. Latest consensus state: current_term: 1 leader_uuid: "5996c5d602674342be1a5b3f18e69d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } } }
I20260924 10:13:25.626585 29261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:25.628120 29272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5996c5d602674342be1a5b3f18e69d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } } }
I20260924 10:13:25.628523 29272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:13:25.639781 29278 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260924 10:13:25.664211 6155 mini_kdc.cc:348] Time spent kdestroy: real 0.102s user 0.002s sys 0.021s
W20260924 10:13:25.671250 29212 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:25.667492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.6.2.254:40545)
0924 10:13:25.667757 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.667764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.668775 (+ 1011us) server_negotiation.cc:407] Connection header received
0924 10:13:25.669135 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.669145 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.669254 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.669367 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.670889 (+ 1522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47338: BlockingRecv error: recv got EOF from 127.0.0.1:47338 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260924 10:13:25.673808 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:25.667975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44120 (local address 127.6.2.253:45977)
0924 10:13:25.668180 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.668188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.671884 (+ 3696us) server_negotiation.cc:407] Connection header received
0924 10:13:25.672179 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.672187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.672288 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.672370 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.673477 (+ 1107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44120: BlockingRecv error: recv got EOF from 127.0.0.1:44120 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 10:13:25.676357 29255 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:25.668834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.6.2.252:40753)
0924 10:13:25.669026 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.669032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.674463 (+ 5431us) server_negotiation.cc:407] Connection header received
0924 10:13:25.674702 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.674710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.674806 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.674879 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.676046 (+ 1167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49070: BlockingRecv error: recv got EOF from 127.0.0.1:49070 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 10:13:25.689002 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:25.682530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.6.2.253:45977)
0924 10:13:25.682768 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.682777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.687580 (+ 4803us) server_negotiation.cc:407] Connection header received
0924 10:13:25.687849 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.687854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.687947 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.688069 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.688785 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44128: BlockingRecv error: recv got EOF from 127.0.0.1:44128 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260924 10:13:25.695264 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:25.693357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.6.2.253:45977)
0924 10:13:25.693525 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.693532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.693865 (+ 333us) server_negotiation.cc:407] Connection header received
0924 10:13:25.694089 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.694093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.694160 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.694268 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.694947 (+ 679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44144: BlockingRecv error: recv got EOF from 127.0.0.1:44144 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 10:13:25.703161 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:25.700732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.6.2.253:45977)
0924 10:13:25.700938 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.700947 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.701465 (+ 518us) server_negotiation.cc:407] Connection header received
0924 10:13:25.701763 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.701768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.701860 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.701933 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.702914 (+ 981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44154: BlockingRecv error: recv got EOF from 127.0.0.1:44154 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 10:13:25.712826 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:25.710628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44158 (local address 127.6.2.253:45977)
0924 10:13:25.710908 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.710917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.711437 (+ 520us) server_negotiation.cc:407] Connection header received
0924 10:13:25.711673 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.711677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.711748 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.711858 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.712586 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44158: BlockingRecv error: recv got EOF from 127.0.0.1:44158 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
W20260924 10:13:25.760893 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:25.758641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44168 (local address 127.6.2.253:45977)
0924 10:13:25.758872 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.758881 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.759459 (+ 578us) server_negotiation.cc:407] Connection header received
0924 10:13:25.759688 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.759699 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.759772 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.759908 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.760690 (+ 782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44168: BlockingRecv error: recv got EOF from 127.0.0.1:44168 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 10:13:25.783422 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:25.781145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.6.2.253:45977)
0924 10:13:25.781409 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.781418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.781950 (+ 532us) server_negotiation.cc:407] Connection header received
0924 10:13:25.782187 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.782200 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.782273 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.782386 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.783147 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44176: BlockingRecv error: recv got EOF from 127.0.0.1:44176 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 10:13:25.821655 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:25.819453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.6.2.253:45977)
0924 10:13:25.819703 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.819713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.820226 (+ 513us) server_negotiation.cc:407] Connection header received
0924 10:13:25.820452 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.820458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.820542 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.820658 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.821418 (+ 760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44182: BlockingRecv error: recv got EOF from 127.0.0.1:44182 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 10:13:25.895591 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:25.893331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.6.2.253:45977)
0924 10:13:25.893584 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:25.893593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:25.894122 (+ 529us) server_negotiation.cc:407] Connection header received
0924 10:13:25.894380 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:25.894384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:25.894458 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:25.894574 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:25.895353 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44198: BlockingRecv error: recv got EOF from 127.0.0.1:44198 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 10:13:26.029452 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:26.027479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.6.2.253:45977)
0924 10:13:26.027699 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:26.027706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:26.028109 (+ 403us) server_negotiation.cc:407] Connection header received
0924 10:13:26.028277 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:26.028282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:26.028375 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:26.028502 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:26.029219 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44208: BlockingRecv error: recv got EOF from 127.0.0.1:44208 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 10:13:26.084182 29231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5996c5d602674342be1a5b3f18e69d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5996c5d602674342be1a5b3f18e69d6f" member_type: VOTER last_known_addr { host: "127.6.2.254" port: 40545 } } peers { permanent_uuid: "7d5d3f99e76a46708089ca2473a45962" member_type: VOTER last_known_addr { host: "127.6.2.253" port: 45977 } } peers { permanent_uuid: "7b66f74555c5460792ef9032a2e38e87" member_type: VOTER last_known_addr { host: "127.6.2.252" port: 40753 } } }
I20260924 10:13:26.084513 29231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5996c5d602674342be1a5b3f18e69d6f [sys.catalog]: This master's current role is: LEADER
W20260924 10:13:26.295428 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:13:26.293433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.6.2.253:45977)
0924 10:13:26.293677 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:26.293683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:26.293864 (+ 181us) server_negotiation.cc:407] Connection header received
0924 10:13:26.294118 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:26.294124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:26.294206 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:26.294322 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:13:26.295153 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44212: BlockingRecv error: recv got EOF from 127.0.0.1:44212 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 10:13:26.364449 29228 catalog_manager.cc:1259] Loaded cluster ID: ac68dcedfcb24dc3953129d393245b1d
I20260924 10:13:26.364552 29228 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87: loading cluster ID for follower catalog manager: success
I20260924 10:13:26.367034 29228 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87: acquiring CA information for follower catalog manager: success
I20260924 10:13:26.368969 29228 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b66f74555c5460792ef9032a2e38e87: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:13:26.644125 29278 catalog_manager.cc:1259] Loaded cluster ID: ac68dcedfcb24dc3953129d393245b1d
I20260924 10:13:26.644316 29278 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962: loading cluster ID for follower catalog manager: success
I20260924 10:13:26.647007 29278 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962: acquiring CA information for follower catalog manager: success
I20260924 10:13:26.649292 29278 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d5d3f99e76a46708089ca2473a45962: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:13:26.819044 29346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:26.811689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.6.2.253:45977)
0924 10:13:26.812497 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:26.812502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:26.812522 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:13:26.812591 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:26.812595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:26.812741 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:26.812863 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:26.813402 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:26.814254 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:26.816273 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:26.816880 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:26.817022 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:26.817627 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:26.818272 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:26.818408 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:26.818590 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 10:13:26.818664 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":563,"threads_started":1}
I20260924 10:13:26.831979 29352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:26.825445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59176 (local address 127.6.2.252:40753)
0924 10:13:26.826027 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:26.826041 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:26.826059 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:13:26.826153 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:26.826157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:26.826307 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:26.826451 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:26.826886 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:26.827718 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:26.829580 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:26.829995 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:26.830103 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:26.830708 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:26.831322 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:26.831444 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:26.831576 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 10:13:26.831644 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":356,"threads_started":1}
I20260924 10:13:26.862641 29358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:26.838400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.6.2.254:40545)
0924 10:13:26.851697 (+ 13297us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:26.851745 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:26.852220 (+ 475us) server_negotiation.cc:407] Connection header received
0924 10:13:26.852381 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:26.852390 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:26.852735 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:26.853019 (+ 284us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:26.854021 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:26.855529 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:26.858730 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:26.859973 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:26.860215 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:26.860987 (+ 772us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:26.861502 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:26.861597 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:26.862110 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 10:13:26.862214 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12267,"thread_start_us":13078,"threads_started":1}
I20260924 10:13:26.866418 29346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:26.838891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.6.2.253:45977)
0924 10:13:26.839093 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:26.839102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:26.858997 (+ 19895us) server_negotiation.cc:407] Connection header received
0924 10:13:26.859389 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:26.859402 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:26.859508 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:26.859612 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:26.860482 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:26.861618 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:26.864176 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:26.864818 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:26.864938 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:26.865551 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:26.865691 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:26.865756 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:26.865986 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 10:13:26.866088 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:13:26.867125 29352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:13:26.851865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.6.2.252:40753)
0924 10:13:26.852042 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:13:26.852048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:13:26.859352 (+ 7304us) server_negotiation.cc:407] Connection header received
0924 10:13:26.859596 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:13:26.859603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:13:26.859712 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:13:26.859802 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:13:26.860722 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:26.861749 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:26.864934 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:13:26.865573 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:13:26.865664 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:13:26.866281 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:13:26.866427 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:13:26.866495 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:13:26.866761 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 10:13:26.866844 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:13:26.867966 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 28992
I20260924 10:13:26.869320 29054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:13:27.016469 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 28992
I20260924 10:13:27.038482 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 29066
I20260924 10:13:27.039637 29128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:13:27.148033 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 29066
I20260924 10:13:27.169272 6155 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu pid 29141
I20260924 10:13:27.170459 29203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 10:13:27.279194 6155 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskq3UYkq/build/asan/bin/kudu with pid 29141
2026-09-24T10:13:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4944 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9401 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (370196 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST